Search

Large Language Models (LLMs) have demonstrated remarkable performance through supervised fine-tuning or in-context learning using gold labels. However, this paradigm is limited by the availability of gold labels, while in certain scenarios, LLMs may need to perform tasks that are too complex for humans to provide such labels. To tackle this challenge, this study explores whether solely utilizing unlabeled data can elicit strong model capabilities. We propose a new paradigm termed zero-to-strong generalization. We iteratively prompt LLMs to annotate unlabeled data and retain high-quality labels by filtering. Surprisingly, we obverse that this iterative process gradually unlocks LLMs' potential on downstream tasks. Our experiments on extensive classification and reasoning tasks confirm the effectiveness of our proposed framework. Our analysis indicates that this paradigm is effective for both in-context learning and fine-tuning, and for various model sizes., Comment: 15 pages

Large Language Models (LLMs) have shown remarkable abilities across various tasks, yet their development has predominantly centered on high-resource languages like English and Chinese, leaving low-resource languages underserved. To address this disparity, we present SeaLLMs 3, the latest iteration of the SeaLLMs model family, tailored for Southeast Asian languages. This region, characterized by its rich linguistic diversity, has lacked adequate language technology support. SeaLLMs 3 aims to bridge this gap by covering a comprehensive range of languages spoken in this region, including English, Chinese, Indonesian, Vietnamese, Thai, Tagalog, Malay, Burmese, Khmer, Lao, Tamil, and Javanese. Leveraging efficient language enhancement techniques and a specially constructed instruction tuning dataset, SeaLLMs 3 significantly reduces training costs while maintaining high performance and versatility. Our model excels in tasks such as world knowledge, mathematical reasoning, translation, and instruction following, achieving state-of-the-art performance among similarly sized models. Additionally, we prioritized safety and reliability by addressing both general and culture-specific considerations and incorporated mechanisms to reduce hallucinations. This work underscores the importance of inclusive AI, showing that advanced LLM capabilities can benefit underserved linguistic and cultural communities.

Large language models (LLMs) have demonstrated multilingual capabilities; yet, they are mostly English-centric due to the imbalanced training corpora. Existing works leverage this phenomenon to improve their multilingual performances through translation, primarily on natural language processing (NLP) tasks. This work extends the evaluation from NLP tasks to real user queries and from English-centric LLMs to non-English-centric LLMs. While translation into English can help improve the performance of multilingual NLP tasks for English-centric LLMs, it may not be optimal for all scenarios. For culture-related tasks that need deep language understanding, prompting in the native language tends to be more promising as it better captures the nuances of culture and language. Our experiments reveal varied behaviors among different LLMs and tasks in the multilingual context. Therefore, we advocate for more comprehensive multilingual evaluation and more efforts toward developing multilingual LLMs beyond English-centric ones., Comment: 19 pages

Hierarchical topic modeling aims to discover latent topics from a corpus and organize them into a hierarchy to understand documents with desirable semantic granularity. However, existing work struggles with producing topic hierarchies of low affinity, rationality, and diversity, which hampers document understanding. To overcome these challenges, we in this paper propose Transport Plan and Context-aware Hierarchical Topic Model (TraCo). Instead of early simple topic dependencies, we propose a transport plan dependency method. It constrains dependencies to ensure their sparsity and balance, and also regularizes topic hierarchy building with them. This improves affinity and diversity of hierarchies. We further propose a context-aware disentangled decoder. Rather than previously entangled decoding, it distributes different semantic granularity to topics at different levels by disentangled decoding. This facilitates the rationality of hierarchies. Experiments on benchmark datasets demonstrate that our method surpasses state-of-the-art baselines, effectively improving the affinity, rationality, and diversity of hierarchical topic modeling with better performance on downstream tasks., Comment: Accepted to AAAI2024 conference. Our code is available at https://github.com/bobxwu/TraCo

Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon the Llama-2 model and further advanced through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM-13b models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate., Comment: Technical report, ACL 2024 DEMO TRACK

Existing solutions to zero-shot text classification either conduct prompting with pre-trained language models, which is sensitive to the choices of templates, or rely on large-scale annotated data of relevant tasks for meta-tuning. In this work, we propose a new paradigm based on self-supervised learning to solve zero-shot text classification tasks by tuning the language models with unlabeled data, called self-supervised tuning. By exploring the inherent structure of free texts, we propose a new learning objective called first sentence prediction to bridge the gap between unlabeled data and text classification tasks. After tuning the model to learn to predict the first sentence in a paragraph based on the rest, the model is able to conduct zero-shot inference on unseen tasks such as topic classification and sentiment analysis. Experimental results show that our model outperforms the state-of-the-art baselines on 7 out of 10 tasks. Moreover, the analysis reveals that our model is less sensitive to the prompt design. Our code and pre-trained models are publicly available at https://github.com/DAMO-NLP-SG/SSTuning ., Comment: Accepted to the Findings of ACL 2023

Cross-lingual topic models have been prevalent for cross-lingual text analysis by revealing aligned latent topics. However, most existing methods suffer from producing repetitive topics that hinder further analysis and performance decline caused by low-coverage dictionaries. In this paper, we propose the Cross-lingual Topic Modeling with Mutual Information (InfoCTM). Instead of the direct alignment in previous work, we propose a topic alignment with mutual information method. This works as a regularization to properly align topics and prevent degenerate topic representations of words, which mitigates the repetitive topic issue. To address the low-coverage dictionary issue, we further propose a cross-lingual vocabulary linking method that finds more linked cross-lingual words for topic alignment beyond the translations of a given dictionary. Extensive experiments on English, Chinese, and Japanese datasets demonstrate that our method outperforms state-of-the-art baselines, producing more coherent, diverse, and well-aligned topics and showing better transferability for cross-lingual classification tasks., Comment: Accepted to AAAI2023 conference. Code is available at https://github.com/BobXWu/InfoCTM

Generally, the vortex structures should be independent of the observers who are moving, especially when their coordinates are non-inertial, which may result in confusions in communications between researchers. The property that not being influenced by the choice of coordinate is called objective. Mostly, researchers would like to gather data in the inertial coordinate because most physical laws are valid in the inertial coordinate. As a result, how to transfer the observations in a non-inertial coordinate to those in an inertial coordinate raises interest. In this paper, methods are provided to obtain velocity and velocity gradient tensor in an inertial coordinate from the non-inertial observations based on one known zero-vorticity point which can be chosen from physical fact such as the point is located in the inviscid region.Further, objective vortex structure can be calculated from the objective velocity or velocity gradient tensor. With Liutex chosen as the indicator of the vortex, two numerical examples are used to test the proposed methods. The results show that the methods perform very well and the relative error is extremely small., Comment: 19 pages, 10 figures, 3 tables

There are several questions with NS, which include: 1. Both symmetric shear terms and stretching terms in strain and stress are coordinate-dependent and thus not Galilean invariant; 2. The physical meaning of both diagonal and off-diagonal elements are not clear, which is coordinate-dependent; 3. It is hard to measure the strain and stress quantitatively, and viscosity is really measured by vorticity not by symmetric strain; 4. There is no vorticity tensor in NS, which plays important role in fluid flow especially for turbulent flow. The new proposed governing equations for fluid dynamics use vorticity tensor only, which is anti-symmetric. The advantages include: 1. Both shear and stress are anti-symmetric, which are Galilean invariant and independent of coordinate rotation; 2. The physical meaning of off diagonal elements is clear, which is anti-symmetric shear stress, 3. Viscosity coefficients are obtained by experiment which uses vorticity, 4. The vorticity term can be further decomposed to rigid rotation and anti-symmetric shear, which are important to turbulence research, 5. The computation cost for viscous term is reduced to half as the diagonal terms are all zero as six elements are reduced to three. Several computational results are made, which clearly demonstrate both NS and new governing equations have exactly same results. As shown below, the new governing equation is identical to NS in mathematics, but the former has lower cost and several advantages mentioned above including possibility to better study turbulent flow. There are reasons to use the new governing equations to replace Navier-Stokes equations. The unique definition and operation of vector and tensor by matrix and matrix operation are also discussed in this paper., Comment: New Discovery, 21 pages, 4766 words, 11 figures

Traditional Cauchy-Stokes decomposition of velocity gradient tensor gives a symmetric and an anti-symmetric subtensors which are called the strain-rate and vorticity tensors. There are two problems with Cauchy-Stokes decomposition. The first one is that the anti-symmetric or vorticity tensor cannot represent the fluid rotation or vortex. The second one is that the symmetric (strain-rate) tensor cannot distinguish the stretching (compression) and shear. The stretching and shear are dependent on the coordinate or are not Galilean invariant. Since vorticity cannot distinguish between the non-rotational shear and the rigid rotation, vorticity has been decomposed to a rigid rotation called Liutex and anti-symmetric shear in our previous work. A Liutex-based principal coordinate was developed and the velocity gradient tensor was decomposed in the principal coordinate as a rigid rotation (Liutex tensor), a pure shear tensor and a stretching (compression) tensor, which is called the principal decomposition. However, the principal decomposition is made at each point which has own principal coordinate different from other points. This paper derives the principal decomposition in the original xyz coordinate system, and, therefore, provides a new tool for fluid kinematics to conduct the velocity gradient tensor decomposition to rigid rotation, pure shear, and stretching (compression) which is Galilean invariant and has clear physical meanings. The new velocity gradient tensor decomposition could become a foundation for new fluid kinematics.

As widely recognized, vortex represents flow rotation. Vortex should have a local rotation axis as its direction and angular speed as its strength. Vorticity vector has been considered the rotation axis, and vorticity magnitude the rotational strength for a long time in classical fluid kinematics. However, this concept cannot stand in viscous flow. This study demonstrates by rigorous mathematical proof that the vorticity vector is not the fluid rotation axis, and vorticity is not the rotation strength. On the other hand, the Liutex vector is mathematically proved as the fluid rotation axis, and the Liutex magnitude is twice the fluid angular speed., Comment: 16 pages and three figures