Search

We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy., Comment: Work in progress

This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a balance between the inference efficiency and embedding quality. The training procedure adheres to the English E5 model recipe, involving contrastive pre-training on 1 billion multilingual text pairs, followed by fine-tuning on a combination of labeled datasets. Additionally, we introduce a new instruction-tuned embedding model, whose performance is on par with state-of-the-art, English-only models of similar sizes. Information regarding the model release can be found at https://github.com/microsoft/unilm/tree/master/e5 ., Comment: 6 pages

Visual fine-tuning has garnered significant attention with the rise of pre-trained vision models. The current prevailing method, full fine-tuning, suffers from the issue of knowledge forgetting as it focuses solely on fitting the downstream training set. In this paper, we propose a novel weight rollback-based fine-tuning method called OLOR (One step Learning, One step Review). OLOR combines fine-tuning with optimizers, incorporating a weight rollback term into the weight update term at each step. This ensures consistency in the weight range of upstream and downstream models, effectively mitigating knowledge forgetting and enhancing fine-tuning performance. In addition, a layer-wise penalty is presented to employ penalty decay and the diversified decay rate to adjust the weight rollback levels of layers for adapting varying downstream tasks. Through extensive experiments on various tasks such as image classification, object detection, semantic segmentation, and instance segmentation, we demonstrate the general applicability and state-of-the-art performance of our proposed OLOR. Code is available at https://github.com/rainbow-xiao/OLOR-AAAI-2024., Comment: Accepted to the 38th Annual AAAI Conference on Artificial Intelligence (AAAI 2024)

The Software Engineering (SE) community has been embracing the open science policy and encouraging researchers to disclose artifacts in their publications. However, the status and trends of artifact practice and quality remain unclear, lacking insights on further improvement. In this paper, we present an empirical study to characterize the research artifacts in SE publications. Specifically, we manually collect 1,487 artifacts from all 2,196 papers published in top-tier SE conferences (ASE, FSE, ICSE, and ISSTA) from 2017 to 2022. We investigate the common practices (e.g., URL location and format, storage websites), maintenance activities (e.g., last update time and URL validity), popularity (e.g., the number of stars on GitHub and characteristics), and quality (e.g., documentation and code smell) of these artifacts. Based on our analysis, we reveal a rise in publications providing artifacts. The usage of Zenodo for sharing artifacts has significantly increased. However, artifacts stored in GitHub tend to receive few stars, indicating a limited influence on real-world SE applications. We summarize the results and provide suggestions to different stakeholders in conjunction with current guidelines., Comment: Accepted by Journal of Systems and Software (JSS 2024). Please include JSS in any citations

The rice stem borer (RSB), Chilo suppressalis (Walker), is an economically important pest of rice in subtropical Asia. Up till present, it remains unknown how pest abundance and parasitoid-mediated biological control are modulated by landscape composition. In this study, 20 rice fields with varying proportion of non-crop habitat in (2000 m radius) landscape sectors were selected in China's Jiangxi province. RSB infestation levels were highest in agriculture-dominated landscapes. Meanwhile, parasitism rate increased with pest pressure but was not related to landscape-level non-crop habitat cover. Landscape-level responses of parasitoids were species-specific and likely modulated by functional traits. The specialist parasitoid Cotesia chilonis (Munakata) responded negatively to non-crop habitat, while positive responses were recorded for the generalists Eriborus sinicus (Holmgren) and Microgaster russata (Haliday). Our work unveils the occurrence of a bottom-up cascade in which spatiotemporal rice cropping patterns shape pest pressure and parasitism dynamics, with more diverse landscapes experiencing lower RSB infestation levels. Non-crop habitats within the landscape matrix equally retained parasitoid species with varying feeding modes and attack strategies. Hence, despite its (globally) inconsistent impacts on biological control, agricultural landscape diversification can directly benefit pest suppression and thus remains a valuable component of the ecological intensification toolbox.

In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks., Comment: Accepted by ACL 2024

Modern search engines are built on a stack of different components, including query understanding, retrieval, multi-stage ranking, and question answering, among others. These components are often optimized and deployed independently. In this paper, we introduce a novel conceptual framework called large search model, which redefines the conventional search stack by unifying search tasks with one large language model (LLM). All tasks are formulated as autoregressive text generation problems, allowing for the customization of tasks through the use of natural language prompts. This proposed framework capitalizes on the strong language understanding and reasoning capabilities of LLMs, offering the potential to enhance search result quality while simultaneously simplifying the existing cumbersome search stack. To substantiate the feasibility of this framework, we present a series of proof-of-concept experiments and discuss the potential challenges associated with implementing this approach within real-world search systems., Comment: SIGIR Forum, Vol. 57 No. 2 - December 2023

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters., Comment: 17 pages, v2 fixes the SummEval numbers

Query-aware webpage snippet extraction is widely used in search engines to help users better understand the content of the returned webpages before clicking. Although important, it is very rarely studied. In this paper, we propose an effective query-aware webpage snippet extraction method named DeepQSE, aiming to select a few sentences which can best summarize the webpage content in the context of input query. DeepQSE first learns query-aware sentence representations for each sentence to capture the fine-grained relevance between query and sentence, and then learns document-aware query-sentence relevance representations for snippet extraction. Since the query and each sentence are jointly modeled in DeepQSE, its online inference may be slow. Thus, we further propose an efficient version of DeepQSE, named Efficient-DeepQSE, which can significantly improve the inference speed of DeepQSE without affecting its performance. The core idea of Efficient-DeepQSE is to decompose the query-aware snippet extraction task into two stages, i.e., a coarse-grained candidate sentence selection stage where sentence representations can be cached, and a fine-grained relevance modeling stage. Experiments on two real-world datasets validate the effectiveness and efficiency of our methods., Comment: Accepted by EMNLP2022

Image representations are a critical building block of computer vision applications. This paper presents the 2nd place solution to the Google Universal Image Embedding Competition, which is part of the ECCV2022 instance-level recognition workshops. We use the instance-level fine-grained image classification method to complete this competition. We focus on data building and processing, model structure, and training strategies. Finally, the solution scored 0.713 on the public leaderboard and 0.709 on the private leaderboard., Comment: 3 pages, 1 figures, Instance-Level Recognition Workshop at ECCV 2022, Google Universal Image Embedding, 2nd place solution

In large-scale retrieval, the lexicon-weighting paradigm, learning weighted sparse representations in vocabulary space, has shown promising results with high quality and low latency. Despite it deeply exploiting the lexicon-representing capability of pre-trained language models, a crucial gap remains between language modeling and lexicon-weighting retrieval -- the former preferring certain or low-entropy words whereas the latter favoring pivot or high-entropy words -- becoming the main barrier to lexicon-weighting performance for large-scale retrieval. To bridge this gap, we propose a brand-new pre-training framework, lexicon-bottlenecked masked autoencoder (LexMAE), to learn importance-aware lexicon representations. Essentially, we present a lexicon-bottlenecked module between a normal language modeling encoder and a weakened decoder, where a continuous bag-of-words bottleneck is constructed to learn a lexicon-importance distribution in an unsupervised fashion. The pre-trained LexMAE is readily transferred to the lexicon-weighting retrieval via fine-tuning. On the ad-hoc retrieval benchmark, MS-Marco, it achieves 42.6% MRR@10 with 45.8 QPS for the passage dataset and 44.4% MRR@100 with 134.8 QPS for the document dataset, by a CPU machine. And LexMAE shows state-of-the-art zero-shot transfer capability on BEIR benchmark with 12 datasets., Comment: Appeared at ICLR 2023

In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost. Our code and model check points are available at https://github.com/microsoft/unilm/tree/master/simlm ., Comment: Accepted to ACL 2023

Communication latency has become one of the determining factors for the performance of parallel clusters. To design low-latency network topologies for high-performance computing clusters, we optimize the diameters, mean path lengths, and bisection widths of circulant topologies. We obtain a series of optimal circulant topologies of size $2^5$ through $2^{10}$ and compare them with torus and hypercube of the same sizes and degrees. We further benchmark on a broad variety of applications including effective bandwidth, FFTE, Graph 500 and NAS parallel benchmarks to compare the optimal circulant topologies and Cartesian products of optimal circulant topologies and fully connected topologies with corresponding torus and hypercube. Simulation results demonstrate superior potentials of the optimal circulant topologies for communication-intensive applications. We also find the strengths of the Cartesian products in exploiting global communication with data traffic patterns of specific applications or internal algorithms.

Safety is of great importance in multi-robot navigation problems. In this paper, we propose a control barrier function (CBF) based optimizer that ensures robot safety with both high probability and flexibility, using only sensor measurement. The optimizer takes action commands from the policy network as initial values and then provides refinement to drive the potentially dangerous ones back into safe regions. With the help of a deep transition model that predicts the evolution of surrounding dynamics and the consequences of different actions, the CBF module can guide the optimization in a reasonable time horizon. We also present a novel joint training framework that improves the cooperation between the Reinforcement Learning (RL) based policy and the CBF-based optimizer both in training and inference procedures by utilizing reward feedback from the CBF module. We observe that the policy using our method can achieve a higher success rate while maintaining the safety of multiple robots in significantly fewer episodes compared with other methods. Experiments are conducted in multiple scenarios both in simulation and the real world, the results demonstrate the effectiveness of our method in maintaining the safety of multi-robot navigation. Code is available at \url{https://github.com/YuxiangCui/MARL-OCBF, Comment: 7 pages, 7 figures. conference

Target following in dynamic pedestrian environments is an important task for mobile robots. However, it is challenging to keep tracking the target while avoiding collisions in crowded environments, especially with only one robot. In this paper, we propose a multi-agent method for an arbitrary number of robots to follow the target in a socially-aware manner using only 2D laser scans. The multi-agent following problem is tackled by utilizing the complementary strengths of both reinforcement learning and potential field, in which the reinforcement learning part handles local interactions while navigating to the goals assigned by the potential field. Specifically, with the help of laser scans in obstacle map representation, the learning-based policy can help the robots avoid collisions with both static obstacles and dynamic obstacles like pedestrians in advance, namely socially aware. While the formation control and goal assignment for each robot is obtained from a target-centered potential field constructed using aggregated state information from all the following robots. Experiments are conducted in multiple settings, including random obstacle distributions and different numbers of robots. Results show that our method works successfully in unseen dynamic environments. The robots can follow the target in a socially compliant manner with only 2D laser scans., Comment: 6 pages, 6 figures, conference

We organize a table of regular graphs with minimal diameters and minimal mean path lengths, large bisection widths and high degrees of symmetries, obtained by enumerations on supercomputers. These optimal graphs, many of which are newly discovered, may find wide applications, for example, in design of network topologies., Comment: add details about automorphism group

We introduced the load-balanced routing algorithms, for interconnection networks resulting from nesting, by considering the pressure of the data forwarding in each node. Benchmarks on a small cluster with various network topologies, and simulations for several larger clusters whose prototypes are too costly to construct, demonstrated substantial gains of communication performance with our routing on these networks over other mainstream routing algorithms., Comment: Parts of the results in this manuscript were presented as a Poster at the SC'19 ACM Student Research Competition.(https://sc19.supercomputing.org/proceedings/src_poster/src_poster_pages/spostg138.html). This version of manuscript has been heavily improved with feedbacks from the Poster review committee and visitors to our Poster during SC'19

Using three supercomputers, we broke a record set in 2011, in the enumeration of non-isomorphic regular graphs by expanding the sequence of A006820 in Online Encyclopedia of Integer Sequences (OEIS), to achieve the number for 4-regular graphs of order 23 as 429,668,180,677,439, while discovering serval optimal regular graphs with minimum average shortest path lengths (ASPL) that can be used as interconnection networks for parallel computers. The number of 4-regular graphs and the optimal graphs, extremely time-consuming to calculate, result from a method we adapt from GENREG, a classical regular graph generator, to fit for supercomputers' strengths of using thousands of processor cores.

We propose that clusters interconnected with network topologies having minimal mean path length will increase their overall performance for a variety of applications. We approach our heuristic by constructing clusters of up to 36 nodes having Dragonfly, torus, ring, Chvatal, Wagner, Bidiakis and several other topologies with minimal mean path lengths and by simulating the performance of 256-node clusters with the same network topologies. The optimal (or sub-optimal) low-latency network topologies are found by minimizing the mean path length of regular graphs. The selected topologies are benchmarked using ping-pong messaging, the MPI collective communications, and the standard parallel applications including effective bandwidth, FFTE, Graph 500 and NAS parallel benchmarks. We established strong correlations between the clusters' performances and the network topologies, especially the mean path lengths, for a wide range of applications. In communication-intensive benchmarks, clusters with optimal network topologies out-perform those with mainstream topologies by several folds. It is striking that a mere adjustment of the network topology suffices to reclaim performance from the same computing hardware.

As a natural refrigerant, CO2 gets more and more attention in recent years. The CO2 system runs at a transcritical cycle. This leads to a high running pressure and high pressure difference for the CO2 compressor. In this paper the single stage and two stage rolling piston compressors with same displacement have been developed for CO2 heat pump water heater. And a comprehensive performance comparison has been made with these two kinds of compressor at different suction pressure, discharge pressure and superheat. With these results, a better understanding for the single stage and two stage CO2 compressors could be got.

As a natural refrigerant, CO2 gets more and more attention in recent years. The CO2 system runs at a transcritical cycle. This leads to a high running pressure and high pressure difference for the CO2 compressor. In this paper the single stage and two stage rolling piston compressors with same displacement have been developed for CO2 heat pump water heater. And a comprehensive performance comparison has been made with these two kinds of compressor at different suction pressure, discharge pressure and superheat. With these results, a better understanding for the single stage and two stage CO2 compressors could be got.

Several directions of the demand of decay data in nuclear astrophysics are discussed for radionuclides near the valley of nuclear stability. The current decay data evaluation results have been presented for a number of radionuclides of astrophysical interest. An extended list of such nuclides is offered for their nuclear characteristics to be further evaluated by the Decay Data Evaluation Project cooperation participants.

Several directions of the demand of decay data in nuclear astrophysics are discussed for radionuclides near the valley of nuclear stability. The current decay data evaluation results have been presented for a number of radionuclides of astrophysical interest. An extended list of such nuclides is offered for their nuclear characteristics to be further evaluated by the Decay Data Evaluation Project cooperation participants.

Ultrafine-grained Ti-6Al-4V alloys were fabricated by high energy ball milling and spark plasma sintering. The effect of ball milling time and interstitial content on the microstructure and properties of sintered compacts was investigated and discussed. The sintered compacts consisted of equiaxed alpha+beta matrixes with average grain sizes of 0.51-0.89 mu m and 2-8% micrometer-sized alpha grains. When the ball milling time increased from 10 to 50 h, the volume fraction of coarse grains was reduced. The improvement of thermal stability may be attributed to the pinning of grain boundaries by nanostructured TiO2 particles and solute drag of interstitial atoms. The sintered compacts with ultrafine-grained structures exhibited 80-120% higher compressive yield strength than that of the coarse-grained alloy. The contributions of grain refinement strengthening and solid-solution/oxide dispersion strengthening via interstitial elements were evaluated by a modified Hall-Petch equation: sigma(cy) = 393+0.46d(-1/2)+5190eq(1/2). When the ball milling time was 10 h, a balance of high strength (compressive yield strength=1260 MPa, ultimate compressive strength=1663 MPa) and sufficient plasticity (plastic strain to failure=20%) could be achieved. (C) 2013 Elsevier B.V. All rights reserved.

Ultrafine-grained Ti-6Al-4V alloys were fabricated by high energy ball milling and spark plasma sintering. The effect of ball milling time and interstitial content on the microstructure and properties of sintered compacts was investigated and discussed. The sintered compacts consisted of equiaxed alpha+beta matrixes with average grain sizes of 0.51-0.89 mu m and 2-8% micrometer-sized alpha grains. When the ball milling time increased from 10 to 50 h, the volume fraction of coarse grains was reduced. The improvement of thermal stability may be attributed to the pinning of grain boundaries by nanostructured TiO2 particles and solute drag of interstitial atoms. The sintered compacts with ultrafine-grained structures exhibited 80-120% higher compressive yield strength than that of the coarse-grained alloy. The contributions of grain refinement strengthening and solid-solution/oxide dispersion strengthening via interstitial elements were evaluated by a modified Hall-Petch equation: sigma(cy) = 393+0.46d(-1/2)+5190eq(1/2). When the ball milling time was 10 h, a balance of high strength (compressive yield strength=1260 MPa, ultimate compressive strength=1663 MPa) and sufficient plasticity (plastic strain to failure=20%) could be achieved. (C) 2013 Elsevier B.V. All rights reserved.

An edge-colored graph $G$ is rainbow connected if any two vertices are connected by a path whose edges have distinct colors. The rainbow connection number of a connected graph $G$, denoted by $rc(G)$, is the smallest number of colors that are needed in order to make $G$ rainbow connected. It was proved that computing $rc(G)$ is an NP-Hard problem, as well as that even deciding whether a graph has $rc(G)=2$ is NP-Complete. It is known that deciding whether a given edge-colored graph is rainbow connected is NP-Complete. We will prove that it is still NP-Complete even when the edge-colored graph is a planar bipartite graph. We also give upper bounds of the rainbow connection number of outerplanar graphs with small diameters. A vertex-colored graph is rainbow vertex-connected if any two vertices are connected by a path whose internal vertices have distinct colors. The rainbow vertex-connection number of a connected graph $G$, denoted by $rvc(G)$, is the smallest number of colors that are needed in order to make $G$ rainbow vertex-connected. It is known that deciding whether a given vertex-colored graph is rainbow vertex-connected is NP-Complete. We will prove that it is still NP-Complete even when the vertex-colored graph is a line graph., Comment: 13 pages

Wireless ad-hoc networks are dynamic multi-hop wireless networks established by a group of mobile stations without pre-existing network infrastructure or centralized administration. Several MAC protocols are devised to address the “hidden terminal” problem in wireless ad-hoc networks. Among them, IEEE 802.11 Distributed Foundation Wireless Medium Access Control (DFWMAC) is a proposed standard for wireless ad-hoc and infrastructure LANs. Although DFWMAC achieves reasonable throughput by alleviating “hidden terminal” problem, it still suffers from a severe fairness problem under heavy traffic, because of the multi-hop nature and its channel access scheme. In this thesis, the architecture with three components is proposed to solve the fairness problem in IEEE 802.11 MAC layer. The deployment of the architecture intends to achieve max-min fairness bandwidth allocation and access among one-hop flows in MAC layer of wireless ad-hoc networks. A distributed Max-min fairshare assignment algorithm is derived to assign max-min fairshare for one-hop flows in wireless ad-hoc networks only, with local flow information. The fairshare computed by the distributed algorithm can converge to the fairshare computed by a centralized administration with global flow information. A flow information dissemination protocol is proposed and implemented to support the distributed algorithm. With the assigned fairshare, an estimation based channel access scheme is proposed to fairly access the channel for one-hop flows. The channel access scheme modifies the backoff scheme in IEEE 802.11, as well as, estimates the traffic of local flows and non-local flows. The channel access scheme achieves fairness among one-hop flows as well as maintains the overall throughput in original IEEE 802.11. Simulations are conducted to show the convergence of the distributed max-min fairshare assignment algorithm and the integrity of the channel access scheme. It was shown that max-min fair shares can be assigned wi

Wireless ad-hoc networks are dynamic multi-hop wireless networks established by a group of mobile stations without pre-existing network infrastructure or centralized administration. Several MAC protocols are devised to address the “hidden terminal” problem in wireless ad-hoc networks. Among them, IEEE 802.11 Distributed Foundation Wireless Medium Access Control (DFWMAC) is a proposed standard for wireless ad-hoc and infrastructure LANs. Although DFWMAC achieves reasonable throughput by alleviating “hidden terminal” problem, it still suffers from a severe fairness problem under heavy traffic, because of the multi-hop nature and its channel access scheme. In this thesis, the architecture with three components is proposed to solve the fairness problem in IEEE 802.11 MAC layer. The deployment of the architecture intends to achieve max-min fairness bandwidth allocation and access among one-hop flows in MAC layer of wireless ad-hoc networks. A distributed Max-min fairshare assignment algorithm is derived to assign max-min fairshare for one-hop flows in wireless ad-hoc networks only, with local flow information. The fairshare computed by the distributed algorithm can converge to the fairshare computed by a centralized administration with global flow information. A flow information dissemination protocol is proposed and implemented to support the distributed algorithm. With the assigned fairshare, an estimation based channel access scheme is proposed to fairly access the channel for one-hop flows. The channel access scheme modifies the backoff scheme in IEEE 802.11, as well as, estimates the traffic of local flows and non-local flows. The channel access scheme achieves fairness among one-hop flows as well as maintains the overall throughput in original IEEE 802.11. Simulations are conducted to show the convergence of the distributed max-min fairshare assignment algorithm and the integrity of the channel access scheme. It was shown that max-min fair shares can be assigned wi