Search

The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC., Comment: This is the preprint version of the paper accepted for presentation at the 2024 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'24). \c{opyright} 2024 IEEE. Personal use of this material is permitted. For other uses, permission from IEEE must be obtained. Please refer to IEEE Xplore for the final published version

We introduce an open-source architecture for next-generation Radio-Access Network baseband processing: 1024 latency-tolerant 32-bit RISC-V cores share 4 MiB of L1 memory via an ultra-low latency interconnect (7-11 cycles), a modular Direct Memory Access engine provides an efficient link to a high bandwidth memory, such as HBM2E (98% peak bandwidth at 910GBps). The system achieves leading-edge energy efficiency at sub-ms latency in key 6G baseband processing kernels: Fast Fourier Transform (93 GOPS/W), Beamforming (125 GOPS/W), Channel Estimation (96 GOPS/W), and Linear System Inversion (61 GOPS/W), with only 9% data movement overhead.

Temporal causal discovery is a crucial task aimed at uncovering the causal relations within time series data. The latest temporal causal discovery methods usually train deep learning models on prediction tasks to uncover the causality between time series. They capture causal relations by analyzing the parameters of some components of the trained models, e.g., attention weights and convolution weights. However, this is an incomplete mapping process from the model parameters to the causality and fails to investigate the other components, e.g., fully connected layers and activation functions, that are also significant for causal discovery. To facilitate the utilization of the whole deep learning models in temporal causal discovery, we proposed an interpretable transformer-based causal discovery model termed CausalFormer, which consists of the causality-aware transformer and the decomposition-based causality detector. The causality-aware transformer learns the causal representation of time series data using a prediction task with the designed multi-kernel causal convolution which aggregates each input time series along the temporal dimension under the temporal priority constraint. Then, the decomposition-based causality detector interprets the global structure of the trained causality-aware transformer with the proposed regression relevance propagation to identify potential causal relations and finally construct the causal graph. Experiments on synthetic, simulated, and real datasets demonstrate the state-of-the-art performance of CausalFormer on discovering temporal causality. Our code is available at https://github.com/lingbai-kong/CausalFormer.

We present Occamy, a 432-core RISC-V dual-chiplet 2.5D system for efficient sparse linear algebra and stencil computations on FP64 and narrow (32-, 16-, 8-bit) SIMD FP data. Occamy features 48 clusters of RISC-V cores with custom extensions, two 64-bit host cores, and a latency-tolerant multi-chiplet interconnect and memory system with 32 GiB of HBM2E. It achieves leading-edge utilization on stencils (83 %), sparse-dense (42 %), and sparse-sparse (49 %) matrix multiply., Comment: 2 pages, 7 figures. Accepted at the 2024 IEEE Symposium on VLSI Technology & Circuits

Over any fixed totally real number field with narrow class number one, we prove that the Rankin-Cohen bracket of two Hecke eigenforms for the Hilbert modular group can only be a Hecke eigenform for dimension reasons, except for a couple of cases where the Rankin-Selberg method does not apply. We shall also prove a conjecture of Freitag on the volume of Hilbert modular groups, and assuming a conjecture of Freitag on the dimension of the cuspform space, we obtain a finiteness result on eigenform product identities.

We generalize the linear relation formula between the square of normalized Hecke eigenforms of weight $k$ and normalized Hecke eigenforms of weight $2k$, to Rankin-Cohen brackets of general degree. As an ingredient of the proof, we also generalize a formula of Zagier on the Petersson inner product of Rankin-Cohen brackets involving Eisenstein series.

Radio Access Networks (RAN) workloads are rapidly scaling up in data processing intensity and throughput as the 5G (and beyond) standards grow in number of antennas and sub-carriers. Offering flexible Processing Elements (PEs), efficient memory access, and a productive parallel programming model, many-core clusters are a well-matched architecture for next-generation software-defined RANs, but staggering performance requirements demand a high number of PEs coupled with extreme Power, Performance and Area (PPA) efficiency. We present the architecture, design, and full physical implementation of Terapool-SDR, a cluster for Software Defined Radio (SDR) with 1024 latency-tolerant, compact RV32 PEs, sharing a global view of a 4MiB, 4096-banked, L1 memory. We report various feasible configurations of TeraPool-SDR featuring an ultra-high bandwidth PE-to-L1-memory interconnect, clocked at 730MHz, 880MHz, and 924MHz (TT/0.80 V/25 {\deg}C) in 12nm FinFET technology. The TeraPool-SDR cluster achieves high energy efficiency on all SDR key kernels for 5G RANs: Fast Fourier Transform (93GOPS/W), Matrix-Multiplication (125GOPS/W), Channel Estimation (96GOPS/W), and Linear System Inversion (61GOPS/W). For all the kernels, it consumes less than 10W, in compliance with industry standards., Comment: 6 pages, 6 figures and 3 tables

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

This paper introduces a novel approach in designing prediction horizons on a generalized predictive control for a DC/DC boost converter. This method involves constructing a closed-loop system model and assessing the impact of different prediction horizons on system stability. In contrast to conventional design approaches that often rely on empirical prediction horizon selection or incorporate non-linear observers, the proposed method establishes a rigorous boundary for the prediction horizon to ensure system stability. This approach facilitates the selection of an appropriate prediction horizon while avoiding excessively short horizons that can lead to instability and preventing the adoption of unnecessarily long horizons that would burden the controller with high computational demands. Finally, the accuracy of the design method has been confirmed through experimental testing. Moreover, it has been demonstrated that the prediction horizon determined by this method reduces the computational burden by 10\%-20\% compared to the empirically selected prediction horizon.

Image segmentation is a clustering task whereby each pixel is assigned a cluster label. Remote sensing data usually consists of multiple bands of spectral images in which there exist semantically meaningful land cover subregions, co-registered with other source data such as LIDAR (LIght Detection And Ranging) data, where available. This suggests that, in order to account for spatial correlation between pixels, a feature vector associated with each pixel may be a vectorized tensor representing the multiple bands and a local patch as appropriate. Similarly, multiple types of texture features based on a pixel's local patch would also be beneficial for encoding locally statistical information and spatial variations, without necessarily labelling pixel-wise a large amount of ground truth, then training a supervised model, which is sometimes impractical. In this work, by resorting to label only a small quantity of pixels, a new semi-supervised segmentation approach is proposed. Initially, over all pixels, an image data matrix is created in high dimensional feature space. Then, t-SNE projects the high dimensional data onto 3D embedding. By using radial basis functions as input features, which use the labelled data samples as centres, to pair with the output class labels, a modified canonical correlation analysis algorithm, referred to as RBF-CCA, is introduced which learns the associated projection matrix via the small labelled data set. The associated canonical variables, obtained for the full image, are applied by k-means clustering algorithm. The proposed semi-supervised RBF-CCA algorithm has been implemented on several remotely sensed multispectral images, demonstrating excellent segmentation results.

Dense Matrix Multiplication (MatMul) is arguably one of the most ubiquitous compute-intensive kernels, spanning linear algebra, DSP, graphics, and machine learning applications. Thus, MatMul optimization is crucial not only in high-performance processors but also in embedded low-power platforms. Several Instruction Set Architectures (ISAs) have recently included matrix extensions to improve MatMul performance and efficiency at the cost of added matrix register files and units. In this paper, we propose Matrix eXtension (MX), a lightweight approach that builds upon the open-source RISC-V Vector (RVV) ISA to boost MatMul energy efficiency. Instead of adding expensive dedicated hardware, MX uses the pre-existing vector register file and functional units to create a hybrid vector/matrix engine at a negligible area cost (< 3%), which comes from a compact near-FPU tile buffer for higher data reuse, and no clock frequency overhead. We implement MX on a compact and highly energy-optimized RVV processor and evaluate it in both a Dual- and 64-Core cluster in a 12-nm technology node. MX boosts the Dual-Core's energy efficiency by 10% for a double-precision 64x64x64 matrix multiplication with the same FPU utilization (~97%) and by 25% on the 64-Core cluster for the same benchmark on 32-bit data, with a 56% performance gain.

The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

We study Hilbert Poincar\'e series associated to general seed functions and construct Cohen's kernels and double Eisenstein series as series of Hilbert Poincar\'e series. Then we calculate the Rankin-Cohen brackets of Hilbert Poincar\'e series and Hilbert modular forms and extend Zagier's kernel formula to totally real number fields. Finally, we show that the Rankin-Cohen brackets of two different types of Eisenstein series are special values of double Eisenstein series up to a constant.

We prove that if $\nu$ has small norm with respect to the level and the weight, the $\nu$-th Hilbert Poincar\'e series does not vanish identically. We also prove Selberg's identity on Kloosterman sums in the case of number fields, which implies certain vanishing and non-vanishing relations of Hilbert Poincar\'e series when the narrow class number is $1$. Finally, we pass to the adelic setting and interpret the problem via Hecke operators.

Synchronization is likely the most critical performance killer in shared-memory parallel programs. With the rise of multi-core and many-core processors, the relative impact on performance and energy overhead of synchronization is bound to grow. This paper focuses on barrier synchronization for TeraPool, a cluster of 1024 RISC-V processors with non-uniform memory access to a tightly coupled 4MB shared L1 data memory. We compare the synchronization strategies available in other multi-core and many-core clusters to identify the optimal native barrier kernel for TeraPool. We benchmark a set of optimized barrier implementations and evaluate their performance in the framework of the widespread fork-join Open-MP style programming model. We test parallel kernels from the signal-processing and telecommunications domain, achieving less than 10% synchronization overhead over the total runtime for problems that fit TeraPool's L1 memory. By fine-tuning our tree barriers, we achieve 1.6x speed-up with respect to a naive central counter barrier and just 6.2% overhead on a typical 5G application, including a challenging multistage synchronization kernel. To our knowledge, this is the first work where shared-memory barriers are used for the synchronization of a thousand processing elements tightly coupled to shared data memory., Comment: 15 pages, 7 figures

Understanding the phase transition mechanisms in two-dimensional (2D) materials is a key to precisely tailor their properties at the nanoscale. Molybdenum ditelluride (MoTe2) exhibits multiple phases at room temperature, making it a promising candidate for phase-change applications. Here, we fabricate lateral 2H-Td interfaces with laser irradiation and probe their phase transitions from micro- to atomic scales with in situ heating in the transmission electron microscope (TEM). By encapsulating the MoTe2 with graphene protection layers, we create an in situ reaction cell compatible with atomic resolution imaging. We find that the Td-to-2H phase transition initiates at phase boundaries at low temperatures (200-225 degree C) and propagates anisotropically along the b-axis in a layer-by-layer fashion. We also demonstrate a fully reversible 2H-Td-2H phase transition cycle, which generates a coherent 2H lattice containing inversion domain boundaries. Our results provide insights on fabricating 2D hetero-phase devices with atomically sharp and coherent interfaces., Comment: 38 pages, 5 figures, 9 supplementary figures, and supplementary information

We first obtain the dimension formulas for the spaces of holomorphic modular forms with character for the Fricke group $\Gamma_0^+(N)$, then that for $\Gamma_0^*(N)$ with all Atkin-Lehner involutions added in a particular case.

5G Radio access network disaggregation and softwarization pose challenges in terms of computational performance to the processing units. At the physical layer level, the baseband processing computational effort is typically offloaded to specialized hardware accelerators. However, the trend toward software-defined radio-access networks demands flexible, programmable architectures. In this paper, we explore the software design, parallelization and optimization of the key kernels of the lower physical layer (PHY) for physical uplink shared channel (PUSCH) reception on MemPool and TeraPool, two manycore systems having respectively 256 and 1024 small and efficient RISC-V cores with a large shared L1 data memory. PUSCH processing is demanding and strictly time-constrained, it represents a challenge for the baseband processors, and it is also common to most of the uplink channels. Our analysis thus generalizes to the entire lower PHY of the uplink receiver at gNodeB (gNB). Based on the evaluation of the computational effort (in multiply-accumulate operations) required by the PUSCH algorithmic stages, we focus on the parallel implementation of the dominant kernels, namely fast Fourier transform, matrix-matrix multiplication, and matrix decomposition kernels for the solution of linear systems. Our optimized parallel kernels achieve respectively on MemPool and TeraPool speedups of 211, 225, 158, and 762, 880, 722, at high utilization (0.81, 0.89, 0.71, and 0.74, 0.88, 0.71), comparable a single-core serial execution, moving a step closer toward a full-software PUSCH implementation., Comment: 6 pages, 9 figures, conference paper

Modern high-performance computing architectures (Multicore, GPU, Manycore) are based on tightly-coupled clusters of processing elements, physically implemented as rectangular tiles. Their size and aspect ratio strongly impact the achievable operating frequency and energy efficiency, but they should be as flexible as possible to achieve a high utilization for the top-level die floorplan. In this paper, we explore the flexibility range for a high-performance cluster of RISC-V cores with shared L1 memory used to build scalable accelerators, with the goal of establishing a hierarchical implementation methodology where clusters can be modeled as soft tiles to achieve optimal die utilization., Comment: 6 pages. Accepted for publication in the IEEE Computer Society Annual Symposium on VLSI (ISVLSI) 2022

By explicitly calculating and then analytically continuing the Fourier expansion of the twisted double Eisenstein series $E_{s,k-s}^{*}(z,w; 1/2)$ of Diamantis and O'Sullivan, we prove a formula of the Petersson inner product of Cohen's kernel and one of its twists, and obtain a rationality result. This extends a result of Kohnen and Zagier.

We analyze a fully discrete finite element numerical scheme for the Cahn-Hilliard-Stokes-Darcy system that models two-phase flows in coupled free flow and porous media. To avoid a well-known difficulty associated with the coupling between the Cahn-Hilliard equation and the fluid motion, we make use of the operator-splitting in the numerical scheme, so that these two solvers are decoupled, which in turn would greatly improve the computational efficiency. The unique solvability and the energy stability have been proved in~\cite{CHW2017}. In this work, we carry out a detailed convergence analysis and error estimate for the fully discrete finite element scheme, so that the optimal rate convergence order is established in the energy norm, i.e.,, in the $\ell^\infty (0, T; H^1) \cap \ell^2 (0, T; H^2)$ norm for the phase variables, as well as in the $\ell^\infty (0, T; H^1) \cap \ell^2 (0, T; H^2)$ norm for the velocity variable. Such an energy norm error estimate leads to a cancellation of a nonlinear error term associated with the convection part, which turns out to be a key step to pass through the analysis. In addition, a discrete $\ell^2 (0;T; H^3)$ bound of the numerical solution for the phase variables plays an important role in the error estimate, which is accomplished via a discrete version of Gagliardo-Nirenberg inequality in the finite element setting.

Multi-task learning (MTL) is an important subject in machine learning and artificial intelligence. Its applications to computer vision, signal processing, and speech recognition are ubiquitous. Although this subject has attracted considerable attention recently, the performance and robustness of the existing models to different tasks have not been well balanced. This article proposes an MTL model based on the architecture of the variational information bottleneck (VIB), which can provide a more effective latent representation of the input features for the downstream tasks. Extensive observations on three public data sets under adversarial attacks show that the proposed model is competitive to the state-of-the-art algorithms concerning the prediction accuracy. Experimental results suggest that combining the VIB and the task-dependent uncertainties is a very effective way to abstract valid information from the input features for accomplishing multiple tasks., Comment: 10 pages

In this paper, we consider the first negative eigenvalue of eigenforms of half-integral weight k + 1/2 and obtain an almost type bound.

A generalized Riemann hypothesis states that all zeros of the completed Hecke $L$-function $L^*(f,s)$ of a normalized Hecke eigenform $f$ on the full modular group should lie on the vertical line $Re(s)=\frac{k}{2}.$ It was shown by Kohnen that there exists a Hecke eigenform $f$ of weight $k$ such that $L^*(f,s) \neq 0$ for sufficiently large $k$ and any point on the line segments $Im(s)=t_0, \frac{k-1}{2} < Re(s) < \frac{k}{2}-\epsilon, \frac{k }{2}+\epsilon < Re(s) < \frac{k+1}{2},$ for any given real number $t_0$ and a positive real number $\epsilon.$ This paper concerns the non-vanishing of the product $L^*(f,s)L^*(f,w)$ $(s,w\in \mathbb{C})$ on average.