Search

The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC., Comment: This is the preprint version of the paper accepted for presentation at the 2024 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'24). \c{opyright} 2024 IEEE. Personal use of this material is permitted. For other uses, permission from IEEE must be obtained. Please refer to IEEE Xplore for the final published version

This work is dedicated to novel uniqueness results and high resolution sampling methods for source support from multi-frequency sparse far field patterns. With a single pair of observation directions $\pm\hat{x}$, we prove that the lines $\{z\in\mathbb R^2|\, \hat{x}\cdot z = \hat{x}\cdot y, \,y\in A_{\hat{x}}\}$ can be determined by multi-frequency far field patterns at the directions $\pm\hat{x}$, where $A_{\hat{x}}$ denotes a set containing the corners of the boundary and points whose normal vector to the boundary is parallel to $\hat{x}$. Furthermore, if the source support is composed of polygons and annuluses, then we prove that the support can be determined by multi-frequency far field patterns at sparse directions. Precisely, the lowest number of the observation directions is given in terms of the number of the corners and the annuluses. Inspired by the uniqueness arguments, we introduce two novel indicators to determine the source support. Numerical examples in two dimensions are presented to show the validity and robustness of the two indicators for reconstructing the boundaries of the source support with a high resolution. The second indicator also shows its powerful ability to determine the unknown source function.

By increasing model parameters but activating them sparsely when performing a task, the use of Mixture-of-Experts (MoE) architecture significantly improves the performance of Large Language Models (LLMs) without increasing the inference cost. However, the memory consumption due to the growing number of experts presents a challenge to the deployment of these models in many real world settings. Our empirical study reveals that some experts encode redundant knowledge during pre-training. We thus propose a method of grouping and pruning similar experts to improve model's parameter efficiency. We validate the effectiveness of our method by pruning two state-of-the-art MoE models, Mixtral-8x7B and Mixtral-8x22B. Evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks. To facilitate future research, we will release our code and the pruned MoE models., Comment: 13pages, 6 figures

Our study delves into Multimodal Entity Linking, aligning the mention in multimodal information with entities in knowledge base. Existing methods are still facing challenges like ambiguous entity representations and limited image information utilization. Thus, we propose dynamic entity extraction using ChatGPT, which dynamically extracts entities and enhances datasets. We also propose a method: Dynamically Integrate Multimodal information with knowledge base (DIM), employing the capability of the Large Language Model (LLM) for visual understanding. The LLM, such as BLIP-2, extracts information relevant to entities in the image, which can facilitate improved extraction of entity features and linking them with the dynamic entity representations provided by ChatGPT. The experiments demonstrate that our proposed DIM method outperforms the majority of existing methods on the three original datasets, and achieves state-of-the-art (SOTA) on the dynamically enhanced datasets (Wiki+, Rich+, Diverse+). For reproducibility, our code and collected datasets are released on \url{https://github.com/season1blue/DIM}., Comment: Published on PRCV24

We consider the inverse cracks scattering problems from the far field patterns with a fixed incident direction. We firstly show that the sound-soft cracks can be uniquely determined by the multi-frequency far field patterns with a fixed incident direction. The proof is based on a low frequency asymptotic analysis of the scattered field. One important feature of the uniqueness result is that the background can even be an unknown inhomogeneous medium. A modified Newton method is then proposed for the numerical reconstruction of the shapes and locations of the cracks. Compared to the classical Newton method, the modified Newton method relaxes the dependence of a good initial guess and can be applied for multiple cracks. Numerical examples in two dimensions are presented to demonstrate the feasibility and effectiveness of the modified Newton method. In particular, the quality of the reconstructions can be greatly improved if we use the measurements properly with two frequencies or two incident directions.

This paper presents a significant improvement on the previous conference paper known as DefSent. The prior study seeks to improve sentence embeddings of language models by projecting definition sentences into the vector space of dictionary entries. We discover that this approach is not fully explored due to the methodological limitation of using word embeddings of language models to represent dictionary entries. This leads to two hindrances. First, dictionary entries are constrained by the single-word vocabulary, and thus cannot be fully exploited. Second, semantic representations of language models are known to be anisotropic, but pre-processing word embeddings for DefSent is not allowed because its weight is frozen during training and tied to the prediction layer. In this paper, we propose a novel method to progressively build entry embeddings not subject to the limitations. As a result, definition sentences can be projected into a quasi-isotropic or isotropic vector space of unlimited dictionary entries, so that sentence embeddings of noticeably better quality are attainable. We abbreviate our approach as DefSent+ (a plus version of DefSent), involving the following strengths: 1) the task performance on measuring sentence similarities is significantly improved compared to DefSent; 2) when DefSent+ is used to further train data-augmented models like SIMCSE, SNCSE, and SynCSE, state-of-the-art performance on measuring sentence similarities can be achieved among the approaches without using manually labeled datasets; 3) DefSent+ is also competitive in feature-based transfer for NLP downstream tasks.

A morphological and photometric analysis of the naked-eye long-period comet C/2022 E3 (ZTF) before perihelion is presented in this study. The observation images taken by the Zwicky Transient Facility survey telescope from July 2022 to October 2022 show a gradually brightening dust coma and a tail with a clear structure. The morphology of the dust coma reveals nonsteady-state emission with an ejection velocity lower than 14 m s$^{-1}$ for particles larger than 100 um. According to the syndyne-synchrone analysis, dust particles larger than about 10 um contribute significantly to the observed tail. The model simulations of the 10 October 2022 image suggest that the radii of large particles lingering near the nucleus range from 0.1 mm to 1 mm. Assuming that the nucleus of comet E3 is a homogeneous sphere with an albedo of 0.1, the photometry analysis sets the lower and upper limits of the nucleus radius to be $0.81\pm0.07$ km and $2.79\pm0.01$ km, respectively. The dust production rates increased continuously from $241\pm3$ kg s$^{-1}$ in July to $476\pm9$ kg s$^{-1}$ in October. The dependence of the ejection velocity $v_{\perp}$ perpendicular to the orbital plane of comet E3 on the particle size $a$ can be simplified as $v_{\perp}\propto a^{-1/2}$, which indicates that the dust emission is likely driven by gas. The water-production rate is inferred as $\sim 368\pm72$ kg s$^{-1}$ in October 2022, which is sustained by an equilibrium-sublimating area of $8.2\times10^6$ m$^2$ at least. The comparative analysis of the characteristics of comet E3 with those of comets belonging to different types shows that the activity profile of long-period comet E3 surprisingly aligns more closely with those of short-period comets within a heliocentric distance range of about [1.7, 3.4] AU, where the images of comet E3 that we used in this study were taken.

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts., Comment: 24 pages

This paper introduces a novel approach to optimize energy efficiency in wavelength division multiplexing (WDM) Visible Light Communication (VLC) systems designed for indoor broadcasting networks. A physics-based LED model is integrated into system energy efficiency optimization, enabling quantitative analysis of the critical issue of VLC energy efficiency: the nonlinear interplay between illumination and communication performance. The optimization jointly incorporates constraints on communication quality of each channel, and illumination performance, standardized by the International Commission on Illumination (CIE). The formulated nonlinear optimization problem is solved by the Sequential Quadratic Programming (SQP) algorithm in an experiment-based simulation. An integrated Red-Green-Blue-Yellow Light Emitting Diode (RGBY-LED) is measured for model calibration and three different scenarios are simulated to evaluate the generality of the proposed method. Results demonstrate a double enhancement in performance and a high versatility in accommodating various scenarios. Furthermore, it highlights the importance of balancing communication and illumination imperatives in VLC systems, challenging conventional perceptions focused solely on minimizing power consumption.

The irregular satellites of Jupiter produce dust particles through the impact of interplanetary micrometeoroids. In this paper, the dynamics of these particles is studied by both high-accuracy numerical simulation and analytical theory, in order to learn their transport, final fate, and spatial distribution. The perturbation forces that are considered in our dynamical model include the solar radiation pressure, solar gravity, Poynting-Robertson drag, Jovian oblateness, and the Galilean satellites' gravity. The trajectories of different size particles are simulated until they hit Jupiter, the Galilean satellites, or escape from the Jovian system. The average dynamical lifetimes of dust with different grain sizes are calculated, and the final fate of dust particles is reported and analysed. The steady-state spatial number density of particles is estimated by integrating the trajectories of dust particles over their initial size distribution, and compared to the previous work. The impact sites of dust on Callisto's surface are recorded and provide an important clue for the study of the hemisphere asymmetry of Callisto. Besides, the mass accretion rate, cross-sectional area influx, and mass influx density of dust on Callisto are calculated. A ring outside the orbit of Callisto dominated by dust between 2 and 25 ${\mu}$m from Jupiter's irregular satellites is suggested, with the average normal geometric optical depth of the order of $10^{-8}$ and the configuration of the ring ansae similar to Jupiter's gossamer rings., Comment: 11 pages, 14 figures

The general capabilities of large language models (LLMs) make them the infrastructure for various AI applications, but updating their inner knowledge requires significant resources. Recent model editing is a promising technique for efficiently updating a small amount of knowledge of LLMs and has attracted much attention. In particular, local editing methods, which directly update model parameters, are more suitable for updating a small amount of knowledge. Local editing methods update weights by computing least squares closed-form solutions and identify edited knowledge by vector-level matching in inference, which achieve promising results. However, these methods still require a lot of time and resources to complete the computation. Moreover, vector-level matching lacks reliability, and such updates disrupt the original organization of the model's parameters. To address these issues, we propose an detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching and adds them to the subject word embeddings in Transformer input. To get these editing embeddings, we propose optimizing then suppressing fusion method, which first optimizes learnable embedding vectors for the editing target and then suppresses the Knowledge Embedding Dimensions (KEDs) to obtain final editing embeddings. We thus propose SWEA$\oplus$OS method for editing factual knowledge in LLMs. We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$\oplus$OS on the \textsc{CounterFact} and zsRE datasets. To further validate the reasoning ability of SWEA$\oplus$OS in editing knowledge, we evaluate it on the more complex \textsc{RippleEdits} benchmark. The results demonstrate that SWEA$\oplus$OS possesses SOTA reasoning ability., Comment: Under review; Our code is available at https://github.com/xpq-tech/SWEA

Large language models (LLMs) often generate convincing, fluent explanations. However, different from humans, they often generate inconsistent explanations on different inputs. For example, an LLM may generate the explanation "all birds can fly" when answering the question "Can sparrows fly?" but meanwhile answer "no" to the related question "Can penguins fly?". Explanations should be consistent across related examples so that they allow a human to simulate the LLM's decision process on multiple examples. We propose explanation-consistency finetuning (EC-finetuning), a method that adapts LLMs to generate more consistent natural-language explanations on related examples. EC-finetuning involves finetuning LLMs on synthetic data that is carefully constructed to contain consistent explanations. Across a variety of question-answering datasets in various domains, EC-finetuning yields a 10.0% relative explanation consistency improvement on four finetuning datasets, and generalizes to seven out-of-distribution datasets not seen during finetuning (+4.5% relative). Code is available at https://github.com/yandachen/explanation-consistency-finetuning ., Comment: arXiv admin note: text overlap with arXiv:2307.08678

The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

Light fidelity (LiFi) is a potential key technology for future 6G networks. However, its feasibility of supporting mobile communications has not been fundamentally discussed. In this paper, we investigate the time-varying channel characteristics of mobile LiFi based on measured mobile phone rotation and movement data. Specifically, we define LiFi channel coherence time to evaluate the correlation of the channel timing sequence. Then, we derive the expression of LiFi transmission rate based on the m-pulse-amplitude-modulation (M-PAM). The derived rate expression indicates that mobile LiFi communications is feasible by using at least two photodiodes (PDs) with different orientations. Further, we propose two channel estimation schemes, and propose a LiFi channel tracking scheme to improve the communication performance. Finally, our experimental results show that the channel coherence time is on the order of tens of milliseconds, which indicates a relatively stable channel. In addition, based on the measured data, better communication performance can be realized in the multiple-input multiple-output (MIMO) scenario with a rate of 36Mbit/s, compared to other scenarios. The results also show that the proposed channel estimation and tracking schemes are effective in designing mobile LiFi systems.

In human-written articles, we often leverage the subtleties of text style, such as bold and italics, to guide the attention of readers. These textual emphases are vital for the readers to grasp the conveyed information. When interacting with large language models (LLMs), we have a similar need - steering the model to pay closer attention to user-specified information, e.g., an instruction. Existing methods, however, are constrained to process plain text and do not support such a mechanism. This motivates us to introduce PASTA - Post-hoc Attention STeering Approach, a method that allows LLMs to read text with user-specified emphasis marks. To this end, PASTA identifies a small subset of attention heads and applies precise attention reweighting on them, directing the model attention to user-specified parts. Like prompting, PASTA is applied at inference time and does not require changing any model parameters. Experiments demonstrate that PASTA can substantially enhance an LLM's ability to follow user instructions or integrate new knowledge from user inputs, leading to a significant performance improvement on a variety of tasks, e.g., an average accuracy improvement of 22% for LLAMA-7B. Our code is publicly available at https://github.com/QingruZhang/PASTA ., Comment: 16 pages

Despite remarkable advancements in mitigating hallucinations in large language models (LLMs) by retrieval augmentation, it remains challenging to measure the reliability of LLMs using static question-answering (QA) data. Specifically, given the potential of data contamination (e.g., leading to memorization), good static benchmark performance does not ensure that model can reliably use the provided evidence for responding, which is essential to avoid hallucination when the required knowledge is new or private. Inspired by adversarial machine learning, we investigate the feasibility of automatically perturbing existing static one for dynamic evaluation. Specifically, this paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases for evaluating the LLMs' reliability in using new evidence for answering. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets on a collection of LLMs under various prompting settings. Our generated data is human-readable and useful to trigger hallucination in LLM. Accurate models on static data are observed to produce unsupported answers from the perturbed evidence, with pronounced accuracy drops across LLMs including GPT-4. We find that our adversarial examples are transferable across all considered LLMs. The examples generated by a small model can be used to evaluate a much larger model, making our approach cost-effective., Comment: NAACL 2024 Findings

The active asteroid 311P is one of the two targets of a planned Chinese asteroid exploration mission Tianwen-2. During 2013, 311P experienced several mass-loss events and exhibited multiple comet-like tails. Here we analyze the morphology and surface brightness of the tails to investigate the dust environment around the nucleus and mechanism of activities. We enhance the features of the tails using image processing techniques to obtain information about the morphology of the tails, and fit processed images to the syndyne-synchrone diagrams. The fitting results give estimations of the upper limits of the durations ($2\sim8$ days) of eruptions and the dust size range ($0.006\sim38.9$ mm) in the tails. The results of surface photometry performed for each dust tail show that the brightness distribution index of each tail ranged from approximately -1.81 to 0 and the dust size distribution indices of 311P's tails ranged from -2.29 to -1.45. The quantity of particles in each tail ranged from 0.5 to 8 $\times10^6$\,kg, which leads to a total dust-loss quantity of $3.0\times10^7$\,kg and a mass loss rate of 1.59 kg s$^{-1}$. Sublimation, continuous impacts or tidal forces of planets are excluded as explanations for 311P's activities, and rotational instability remains a possible activation cause without strong evidence against it., Comment: 18 pages, 8 figures, 3 tables

ELECTRA pre-trains language models by detecting tokens in a sequence that have been replaced by an auxiliary model. Although ELECTRA offers a significant boost in efficiency, its potential is constrained by the training cost brought by the auxiliary model. Notably, this model, which is jointly trained with the main model, only serves to assist the training of the main model and is discarded post-training. This results in a substantial amount of training cost being expended in vain. To mitigate this issue, we propose Fast-ELECTRA, which leverages an existing language model as the auxiliary model. To construct a learning curriculum for the main model, we smooth its output distribution via temperature scaling following a descending schedule. Our approach rivals the performance of state-of-the-art ELECTRA-style pre-training methods, while significantly eliminating the computation and memory cost brought by the joint training of the auxiliary model. Our method also reduces the sensitivity to hyper-parameters and enhances the pre-training stability.