Search

Large language models (LLMs) represented by GPT family have achieved remarkable success. The characteristics of LLMs lie in their ability to accommodate a wide range of tasks through a generative approach. However, the flexibility of their output format poses challenges in controlling and harnessing the model's outputs, thereby constraining the application of LLMs in various domains. In this work, we present Sketch, an innovative toolkit designed to streamline LLM operations across diverse fields. Sketch comprises the following components: (1) a suite of task description schemas and prompt templates encompassing various NLP tasks; (2) a user-friendly, interactive process for building structured output LLM services tailored to various NLP tasks; (3) an open-source dataset for output format control, along with tools for dataset construction; and (4) an open-source model based on LLaMA3-8B-Instruct that adeptly comprehends and adheres to output formatting instructions. We anticipate this initiative to bring considerable convenience to LLM users, achieving the goal of ''plug-and-play'' for various applications. The components of Sketch will be progressively open-sourced at https://github.com/cofe-ai/Sketch.

Controlling the format of outputs generated by large language models (LLMs) is a critical functionality in various applications. Current methods typically employ constrained decoding with rule-based automata or fine-tuning with manually crafted format instructions, both of which struggle with open-domain format requirements. To address this limitation, we introduce a novel framework for controlled generation in LLMs, leveraging user-provided, one-shot QA pairs. This study investigates LLMs' capabilities to follow open-domain, one-shot constraints and replicate the format of the example answers. We observe that this is a non-trivial problem for current LLMs. We also develop a dataset collection methodology for supervised fine-tuning that enhances the open-domain format control of LLMs without degrading output quality, as well as a benchmark on which we evaluate both the helpfulness and format correctness of LLM outputs. The resulting datasets, named OIFC-SFT, along with the related code, will be made publicly available at https://github.com/cofe-ai/OIFC., Comment: 6 pages

Root Cause Analysis (RCA) aims at identifying the underlying causes of system faults by uncovering and analyzing the causal structure from complex systems. It has been widely used in many application domains. Reliable diagnostic conclusions are of great importance in mitigating system failures and financial losses. However, previous studies implicitly assume a full observation of the system, which neglect the effect of partial observation (i.e., missing nodes and latent malfunction). As a result, they fail in deriving reliable RCA results. In this paper, we unveil the issues of unobserved confounders and heterogeneity in partial observation and come up with a new problem of root cause analysis with partially observed data. To achieve this, we propose PORCA, a novel RCA framework which can explore reliable root causes under both unobserved confounders and unobserved heterogeneity. PORCA leverages magnified score-based causal discovery to efficiently optimize acyclic directed mixed graph under unobserved confounders. In addition, we also develop a heterogeneity-aware scheduling strategy to provide adaptive sample weights. Extensive experimental results on one synthetic and two real-world datasets demonstrate the effectiveness and superiority of the proposed framework.

Tool learning methods have enhanced the ability of large language models (LLMs) to interact with real-world applications. Many existing works fine-tune LLMs or design prompts to enable LLMs to select appropriate tools and correctly invoke them to meet user requirements. However, it is observed in previous works that the performance of tool learning varies from tasks, datasets, training settings, and algorithms. Without understanding the impact of these factors, it can lead to inconsistent results, inefficient model deployment, and suboptimal tool utilization, ultimately hindering the practical integration and scalability of LLMs in real-world scenarios. Therefore, in this paper, we explore the impact of both internal and external factors on the performance of tool learning frameworks. Through extensive experiments on two benchmark datasets, we find several insightful conclusions for future work, including the observation that LLMs can benefit significantly from increased trial and exploration. We believe our empirical study provides a new perspective for future tool learning research., Comment: 19 pages, 9 figures

Bent functions are maximally nonlinear Boolean functions with an even number of variables, which include a subclass of functions, the so-called hyper-bent functions whose properties are stronger than bent functions and a complete classification of hyper-bent functions is elusive and inavailable.~In this paper,~we solve an open problem of Mesnager that describes hyper-bentness of hyper-bent functions with multiple trace terms via Dillon-like exponents with coefficients in the extension field~$\mathbb{F}_{2^{2m}}$~of this field~$\mathbb{F}_{2^{m}}$. By applying M\"{o}bius transformation and the theorems of hyperelliptic curves, hyper-bentness of these functions are successfully characterized in this field~$\mathbb{F}_{2^{2m}}$ with~$m$~odd integer., Comment: 10 pages

The inverse scattering transform for the defocusing-defocusing coupled Hirota equations with non-zero boundary conditions at infinity is thoroughly discussed. We delve into the analytical properties of the Jost eigenfunctions and scrutinize the characteristics of the scattering coefficients. To enhance our investigation of the fundamental eigenfunctions, we have derived additional auxiliary eigenfunctions with the help of the adjoint problem. Two symmetry conditions are studied to constrain the behavior of the eigenfunctions and scattering coefficients. Utilizing these symmetries, we precisely delineate the discrete spectrum and establish the associated symmetries of the scattering data. By framing the inverse problem within the context of the Riemann-Hilbert problem, we develop suitable jump conditions to express the eigenfunctions. Consequently, we deduce the pure soliton solutions from the defocusing-defocusing coupled Hirota equations, and the double-poles solutions are provided explicitly for the first time in this work.

With their high-fidelity scene representation capability, the attention of SLAM field is deeply attracted by the Neural Radiation Field (NeRF) and 3D Gaussian Splatting (3DGS). Recently, there has been a surge in NeRF-based SLAM, while 3DGS-based SLAM is sparse. A novel 3DGS-based SLAM approach with a fusion of deep visual feature, dual keyframe selection and 3DGS is presented in this paper. Compared with the existing methods, the proposed tracking is achieved by feature extraction and motion filter on each frame. The joint optimization of poses and 3D Gaussians runs through the entire mapping process. Additionally, the coarse-to-fine pose estimation and compact Gaussian scene representation are implemented by dual keyframe selection and novel loss functions. Experimental results demonstrate that the proposed algorithm not only outperforms the existing methods in tracking and mapping, but also has less memory usage.

Shallow water waves phenomena in nature attract the attention of scholars and play an important role in fields such as tsunamis, tidal waves, solitary waves, and hydraulic engineering. Hereby, fortheshallowwaterwavesphenomenainvariousnaturalenvironments, westudytheKdV-Calogero-Bogoyavlenskii-Schiff (KdV-CBS) equation. Based on the binary Bell polynomial theory, a new general bilinear B\"acklund transformation, Lax pair and infinite conservation laws of the KdV-CBS equation are derived, and it is proved that it is completely integrable in Lax pair sense. Various types of mixed solutions are constructed by using a combination of Homoclinic test method and symbolic computations. These findings have important significance for the discipline, offering vital insights into the intricate dynamics of the KdV-CBS equation. We hope that our research results could help the researchers understand the nonlinear complex phenomena of the shallow water waves in oceans, rivers and coastal areas. Furthermore, the present work can be directly applied to other nonlinear equations.

Recommendation systems play a crucial role in various domains, suggesting items based on user behavior.However, the lack of transparency in presenting recommendations can lead to user confusion. In this paper, we introduce Data-level Recommendation Explanation (DRE), a non-intrusive explanation framework for black-box recommendation models.Different from existing methods, DRE does not require any intermediary representations of the recommendation model or latent alignment training, mitigating potential performance issues.We propose a data-level alignment method, leveraging large language models to reason relationships between user data and recommended items.Additionally, we address the challenge of enriching the details of the explanation by introducing target-aware user preference distillation, utilizing item reviews. Experimental results on benchmark datasets demonstrate the effectiveness of the DRE in providing accurate and user-centric explanations, enhancing user engagement with recommended item., Comment: 5 pages, 2 figures

The challenge of solving the initial value problem for the coupled Lakshmanan Porsezian Daniel equation, while considering nonzero boundary conditions at infinity, is addressed through the development of a suitable inverse scattering transform. Analytical properties of the Jost eigenfunctions are examined, along with the analysis of scattering coefficient characteristics. This analysis leads to the derivation of additional auxiliary eigenfunctions necessary for the comprehensive investigation of the fundamental eigenfunctions. Two symmetry conditions are discussed to study the eigenfunctions and scattering coefficients. These symmetry results are utilized to rigorously define the discrete spectrum and ascertain the corresponding symmetries of scattering datas. The inverse scattering problem is formulated by the Riemann-Hilbert problem. Then we can derive the exact solutions by coupled Lakshmanan Porsezian Daniel equation, the novel soliton solutions are derived and examined in detail., Comment: The article consists of 29 pages and 10 images

Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. However, not all requests posed to LLMs are equally difficult to handle. Through analysis, we show that for some tasks, LLMs can achieve results comparable to the final output at some intermediate layers. That is, not all layers of LLMs are necessary during inference. If we can predict at which layer the inferred results match the final results (produced by evaluating all layers), we could significantly reduce the inference cost. To this end, we propose a simple yet effective algorithm named AdaInfer to adaptively terminate the inference process for an input instance. AdaInfer relies on easily obtainable statistical features and classic classifiers like SVM. Experiments on well-known LLMs like the Llama2 series and OPT, show that AdaInfer can achieve an average of 17.8% pruning ratio, and up to 43% on sentiment tasks, with nearly no performance drop (<1%). Because AdaInfer does not alter LLM parameters, the LLMs incorporated with AdaInfer maintain generalizability across tasks.

Semantic scene completion (SSC) has recently gained popularity because it can provide both semantic and geometric information that can be used directly for autonomous vehicle navigation. However, there are still challenges to overcome. SSC is often hampered by occlusion and short-range perception due to sensor limitations, which can pose safety risks. This paper proposes a fundamental solution to this problem by leveraging vehicle-to-vehicle (V2V) communication. We propose the first generalized collaborative SSC framework that allows autonomous vehicles to share sensing information from different sensor views to jointly perform SSC tasks. To validate the proposed framework, we further build V2VSSC, the first V2V SSC benchmark, on top of the large-scale V2V perception dataset OPV2V. Extensive experiments demonstrate that by leveraging V2V communication, the SSC performance can be increased by 8.3% on geometric metric IoU and 6.0% mIOU.

The paper aims to apply the inverse scattering transform to the defocusing Hirota equation with fully asymmetric non-zero boundary conditions (NZBCs), addressing scenarios in which the solution's limiting values at spatial infinities exhibit distinct non-zero moduli. In comparison to the symmetric case, we explore the characteristic branched nature of the relevant scattering problem explicitly, instead of introducing Riemann surfaces. For the direct problem, we formulate the Jost solutions and scattering data on a single sheet of the scattering variables. We then derive their analyticity behavior, symmetry properties, and the distribution of discrete spectrum. Additionally, we study the behavior of the eigenfunctions and scattering data at the branch points. Finally, the solutions to the defocusing Hirota equation with asymmetric NZBCs are presented through the related Riemann-Hilbert problem on an open contour. Our results can be applicable to the study of asymmetric conditions in nonlinear optics.

Spatio-temporal forecasting of future values of spatially correlated time series is important across many cyber-physical systems (CPS). Recent studies offer evidence that the use of graph neural networks to capture latent correlations between time series holds a potential for enhanced forecasting. However, most existing methods rely on pre-defined or self-learning graphs, which are either static or unintentionally dynamic, and thus cannot model the time-varying correlations that exhibit trends and periodicities caused by the regularity of the underlying processes in CPS. To tackle such limitation, we propose Time-aware Graph Structure Learning (TagSL), which extracts time-aware correlations among time series by measuring the interaction of node and time representations in high-dimensional spaces. Notably, we introduce time discrepancy learning that utilizes contrastive learning with distance-based regularization terms to constrain learned spatial correlations to a trend sequence. Additionally, we propose a periodic discriminant function to enable the capture of periodic changes from the state of nodes. Next, we present a Graph Convolution-based Gated Recurrent Unit (GCGRU) that jointly captures spatial and temporal dependencies while learning time-aware and node-specific patterns. Finally, we introduce a unified framework named Time-aware Graph Convolutional Recurrent Network (TGCRN), combining TagSL, and GCGRU in an encoder-decoder architecture for multi-step spatio-temporal forecasting. We report on experiments with TGCRN and popular existing approaches on five real-world datasets, thus providing evidence that TGCRN is capable of advancing the state-of-the-art. We also cover a detailed ablation study and visualization analysis, offering detailed insight into the effectiveness of time-aware structure learning., Comment: published in ICDE 2024

We propose a theoretical approach based on (constrained) density functional theory and the Franck-Condon approximation for the calculation of the temperature dependent photoluminescence of nanostructures. The method is computationally advantageous and only slightly more demanding than a standard density functional theory calculation and includes transitions into multiphonon final states (higher class transitions). We use the approach for Si and CdSe colloidal nanoclusters (NCs) with up to 693 atoms and obtain very good agreement with experiment which allows us to identify specific peaks and explain their origin. Generally, breathing type modes are shown to dominate the phonon replicas, while optical modes have significant contributions for CdSe NCs and play a lesser role in Si NCs. We obtain significant anti-Stokes peak starting at 140K for Si NC explaining the broadening observed in the corresponding experiment. We also apply the method to small molecular-like carbon structures (diamondoids), where electron-phonon coupling is typically large, and find that multiphonon processes (up to class 4) are very relevant and necessary to compare favorably with experiment. While it is crucial to include these multiphonon states in the small diamondoids with few tens of atoms, neglecting them in only marginally larger Si$_{87}$H$_{76}$ and Cd$_{43}$Se$_{44}$H$^*_{76}$ (and larger) quantum dots represents a good approximation., Comment: Corrected version of Physical Review B 106, 245404 (2022). An Erratum of this article will be published shortly with PRB as well. This pdf is provided for convenience, please cite the original PRB reference including the erratum reference

Large language models (LLMs) have achieved remarkable success in NLP and multimodal tasks, among others. Despite these successes, two main challenges remain in developing LLMs: (i) high computational cost, and (ii) fair and objective evaluations. In this paper, we report a solution to significantly reduce LLM training cost through a growth strategy. We demonstrate that a 101B-parameter LLM with 0.31T tokens can be trained with a budget of 100K US dollars. Inspired by IQ tests, we also consolidate an additional range of evaluations on top of existing evaluations that focus on knowledge-oriented abilities. These IQ evaluations include symbolic mapping, rule understanding, pattern mining, and anti-interference. Such evaluations minimize the potential impact of memorization. Experimental results show that our model, named FLM-101B, trained with a budget of 100K US dollars, achieves performance comparable to powerful and well-known models, e.g., GPT-3 and GLM-130B, especially on the additional range of IQ evaluations. The checkpoint of FLM-101B is released at https://huggingface.co/CofeAI/FLM-101B.

As language models scale up, it becomes increasingly expensive to verify research ideas because conclusions on small models do not trivially transfer to large ones. A possible solution is to establish a generic system that accurately predicts certain metrics for large models without training them. Existing scaling laws require hyperparameter search on the largest models, limiting their predicative capability. In this paper, we present an approach (namely {\mu}Scaling) to predict the pre-training loss, based on our observations that Maximal Update Parametrization ({\mu}P) enables accurate fitting of scaling laws close to common loss basins in hyperparameter space. With {\mu}Scaling, different model designs can be compared on large scales by training only their smaller counterparts. Further, we introduce nanoLM: an affordable LLM pre-training benchmark that facilitates this new research paradigm. With around 14% of the one-time pre-training cost, we can accurately forecast the loss for models up to 52B. Our goal with nanoLM is to empower researchers with limited resources to reach meaningful conclusions on large models. We also aspire for our benchmark to serve as a bridge between the academic community and the industry. Code for {\mu}Scaling is available at https://github.com/cofe-ai/Mu-scaling. Code for nanoLLM will be available later., Comment: This is a modified and extended version of our previous Mu-scaling work released in April 2023 (see v1)

This work applies a combined approach a reduced-order model (ROM) together with experiments and direct numerical simulations to investigate the optimal efficiency of fluid-flow energy harvesting from transverse vortex-induced vibration (VIV) of a circular cylinder. High resolution efficiency maps were predicted over wide ranges of flow reduced velocities and structural damping ratios, and the maximum efficiency and optimal settings of damping ratio and reduced velocity were then examined for different mass ratios and Reynolds numbers. Efficiencies predicted by the ROM were also validated against either experiments or direct simulations. The present work indicates that: (i) the maximum efficiency is controlled by both the incoming reduced velocity and the product of mass ratio and structural damping ratio, which is similar to the maximum amplitude of VIV; (ii) the maximum efficiency at a relatively high Reynolds number ($Re \approx 6 \times 10^3$) in subcritical regime is higher than that of a low Reynolds number ($Re = 150$) in laminar regime; (iii) the energy harvesting efficiency from VIV of a circular cylinder with a low mass ratio is more robust than that with a high mass ratio. This finding suggests that the VIV harvester performs better in water than in air.