Search

The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC., Comment: This is the preprint version of the paper accepted for presentation at the 2024 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'24). \c{opyright} 2024 IEEE. Personal use of this material is permitted. For other uses, permission from IEEE must be obtained. Please refer to IEEE Xplore for the final published version

Diffusion models have shown tremendous results in image generation. However, due to the iterative nature of the diffusion process and its reliance on classifier-free guidance, inference times are slow. In this paper, we propose a new distillation approach for guided diffusion models in which an external lightweight guide model is trained while the original text-to-image model remains frozen. We show that our method reduces the inference computation of classifier-free guided latent-space diffusion models by almost half, and only requires 1\% trainable parameters of the base model. Furthermore, once trained, our guide model can be applied to various fine-tuned, domain-specific versions of the base diffusion model without the need for additional training: this "plug-and-play" functionality drastically improves inference computation while maintaining the visual fidelity of generated images. Empirically, we show that our approach is able to produce visually appealing results and achieve a comparable FID score to the teacher with as few as 8 to 16 steps., Comment: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024 project page: https://5410tiffany.github.io/plug-and-play-diffusion-distillation.github.io/

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

The ultrarelativistic jets triggered by neutrino annihilation processes or Blandford-Znajek (BZ) mechanisms in stellar-mass black hole (BH) hyperaccretion systems are generally considered to power gamma-ray bursts (GRBs). Due to the high accretion rate, the central BHs might grow rapidly on a short timescale, providing a new way to understand "the lower mass gap" problem. In this paper, we use the BH hyperaccretion model to investigate BH mass growth based on observational GRB data. The results show that (i) if the initial BH mass is set as $3~M_\odot$, the neutrino annihilation processes are capable of fueling the BHs to escape the lower mass gap for more than half of long-duration GRBs (LGRBs), while the BZ mechanism is inefficient on triggering BH growths for LGRBs; (ii) the mean BH mass growths in the case of LGRBs without observable supernova (SN) association are much larger than these in the case of LGRBs associated with SNe for both mechanisms, which imply that more massive progenitors or lower SN explosion energies prevail throughout the former cases; (iii) for the short-duration GRBs, the mean BH mass growths are satisfied with the mass supply limitation in the scenario of compact object mergers, but the hyperaccretion processes are unable to rescue BHs from the gap in binary neutron star (NS) mergers or the initial BH mass being $3~M_\odot$ after NS-BH mergers., Comment: 10 pages, 2 figures, 1 table, accepted for publication in ApJ

High-precision surface roughness estimation plays an important role in many applications. However, the classical estimating methods are limited by shot noise and only can achieve the precision of 0.1 nm with white light interferometer. Here, we propose two weak measurement schemes to estimate surface roughness through spectrum analysis and intensity analysis. The estimating precision with spectrum analysis is about $10 ^{-5}$ nm by using a currently available spectrometer with the resolution of $\Delta \lambda= 0.04$ pm and the corresponding sensitivity is better than 0.1 THz/nm. And the precision and sensitivity of the light intensity analysis scheme achieve as high as 0.07 nm and 1/nm, respectively. By introducing a modulated phase, we show that the sensitivity and precision achieved in our schemes can be effectively retained in a wider dynamic range. We further provide the experimental design of the surface profiler based on our schemes. It simultaneously meets the requirements of high precision, high sensitivity, and wide measurement range, making it to be a promising practical tool., Comment: 7 pages, 6 figures

Multi-modality images have been widely used and provide comprehensive information for medical image analysis. However, acquiring all modalities among all institutes is costly and often impossible in clinical settings. To leverage more comprehensive multi-modality information, we propose a privacy secured decentralized multi-modality adaptive learning architecture named ModalityBank. Our method could learn a set of effective domain-specific modulation parameters plugged into a common domain-agnostic network. We demonstrate by switching different sets of configurations, the generator could output high-quality images for a specific modality. Our method could also complete the missing modalities across all data centers, thus could be used for modality completion purposes. The downstream task trained from the synthesized multi-modality samples could achieve higher performance than learning from one real data center and achieve close-to-real performance compare with all real images., Comment: arXiv admin note: substantial text overlap with arXiv:2012.08604

Instance segmentation is of great importance for many biological applications, such as study of neural cell interactions, plant phenotyping, and quantitatively measuring how cells react to drug treatment. In this paper, we propose a novel box-based instance segmentation method. Box-based instance segmentation methods capture objects via bounding boxes and then perform individual segmentation within each bounding box region. However, existing methods can hardly differentiate the target from its neighboring objects within the same bounding box region due to their similar textures and low-contrast boundaries. To deal with this problem, in this paper, we propose an object-guided instance segmentation method. Our method first detects the center points of the objects, from which the bounding box parameters are then predicted. To perform segmentation, an object-guided coarse-to-fine segmentation branch is built along with the detection branch. The segmentation branch reuses the object features as guidance to separate target object from the neighboring ones within the same bounding box region. To further improve the segmentation quality, we design an auxiliary feature refinement module that densely samples and refines point-wise features in the boundary regions. Experimental results on three biological image datasets demonstrate the advantages of our method. The code will be available at https://github.com/yijingru/ObjGuided-Instance-Segmentation., Comment: Accepted in TMI

The sequential quantum random access code (QRAC) allows two or more decoders to obtain a desired message with higher success probability than the best classical bounds by appropriately modulating the measurement sharpness. Here, we propose an entanglement-assisted sequential QRAC protocol which can enable device-independent tasks. By relaxing the equal sharpness and mutually unbiased measurement limits, we widen the sharpness modulation region from a one-dimensional interval to a two-dimensional triangle. Then, we demonstrate our scheme experimentally and get more than 27 standard deviations above the classical bound even when both decoders perform approximately projective measurements. We use the observed success probability to quantify the connection among sequential QRAC, measurement sharpness, measurement biasedness, and measurement incompatibility. Finally, we show that our protocol can be applied to sequential device-independent randomness expansion and our measurement strategy can enhance the success probability of decoding the entire input string. Our results may promote a deeper understanding of the relationship among quantum correlation, quantum measurement, and quantum information processing., Comment: 10 pages, 4 figures, 8 tables

Non-alcoholic fatty liver disease (NAFLD) is one of the most common causes of chronic liver diseases (CLD) which can progress to liver cancer. The severity and treatment of NAFLD is determined by NAFLD Activity Scores (NAS)and liver fibrosis stage, which are usually obtained from liver biopsy. However, biopsy is invasive in nature and involves risk of procedural complications. Current methods to predict the fibrosis and NAS scores from noninvasive CT images rely heavily on either a large annotated dataset or transfer learning using pretrained networks. However, the availability of a large annotated dataset cannot be always ensured andthere can be domain shifts when using transfer learning. In this work, we propose a self-supervised learning method to address both problems. As the NAFLD causes changes in the liver texture, we also propose to use texture encoded inputs to improve the performance of the model. Given a relatively small dataset with 30 patients, we employ a self-supervised network which achieves better performance than a network trained via transfer learning. The code is publicly available at https://github.com/ananyajana/fibrosis_code., Comment: 5 pages, 2 figures, accepted at ISBI 2021, code at this URL: https://github.com/ananyajana/fibrosis_code

Recently, Generative Adversarial Networks (GANs) have demonstrated their potential in federated learning, i.e., learning a centralized model from data privately hosted by multiple sites. A federatedGAN jointly trains a centralized generator and multiple private discriminators hosted at different sites. A major theoretical challenge for the federated GAN is the heterogeneity of the local data distributions. Traditional approaches cannot guarantee to learn the target distribution, which isa mixture of the highly different local distributions. This paper tackles this theoretical challenge, and for the first time, provides a provably correct framework for federated GAN. We propose a new approach called Universal Aggregation, which simulates a centralized discriminator via carefully aggregating the mixture of all private discriminators. We prove that a generator trained with this simulated centralized discriminator can learn the desired target distribution. Through synthetic and real datasets, we show that our method can learn the mixture of largely different distributions where existing federated GAN methods fail.

As deep learning technologies advance, increasingly more data is necessary to generate general and robust models for various tasks. In the medical domain, however, large-scale and multi-parties data training and analyses are infeasible due to the privacy and data security concerns. In this paper, we propose an extendable and elastic learning framework to preserve privacy and security while enabling collaborative learning with efficient communication. The proposed framework is named distributed Asynchronized Discriminator Generative Adversarial Networks (AsynDGAN), which consists of a centralized generator and multiple distributed discriminators. The advantages of our proposed framework are five-fold: 1) the central generator could learn the real data distribution from multiple datasets implicitly without sharing the image data; 2) the framework is applicable for single-modality or multi-modality data; 3) the learned generator can be used to synthesize samples for down-stream learning tasks to achieve close-to-real performance as using actual samples collected from multiple data centers; 4) the synthetic samples can also be used to augment data or complete missing modalities for one single data center; 5) the learning process is more efficient and requires lower bandwidth than other distributed deep learning methods., Comment: arXiv admin note: text overlap with arXiv:2006.00080

Non-Alcoholic Fatty Liver Disease (NAFLD) is becoming increasingly prevalent in the world population. Without diagnosis at the right time, NAFLD can lead to non-alcoholic steatohepatitis (NASH) and subsequent liver damage. The diagnosis and treatment of NAFLD depend on the NAFLD activity score (NAS) and the liver fibrosis stage, which are usually evaluated from liver biopsies by pathologists. In this work, we propose a novel method to automatically predict NAS score and fibrosis stage from CT data that is non-invasive and inexpensive to obtain compared with liver biopsy. We also present a method to combine the information from CT and H\&E stained pathology data to improve the performance of NAS score and fibrosis stage prediction, when both types of data are available. This is of great value to assist the pathologists in computer-aided diagnosis process. Experiments on a 30-patient dataset illustrate the effectiveness of our method., Comment: 6 pages, 3 figures. Accepted in IEEE BIBE 2020

The accurate reconstruction of under-sampled magnetic resonance imaging (MRI) data using modern deep learning technology, requires significant effort to design the necessary complex neural network architectures. The cascaded network architecture for MRI reconstruction has been widely used, while it suffers from the "vanishing gradient" problem when the network becomes deep. In addition, homogeneous architecture degrades the representation capacity of the network. In this work, we present an enhanced MRI reconstruction network using a residual in residual basic block. For each cell in the basic block, we use the differentiable neural architecture search (NAS) technique to automatically choose the optimal operation among eight variants of the dense block. This new heterogeneous network is evaluated on two publicly available datasets and outperforms all current state-of-the-art methods, which demonstrates the effectiveness of our proposed method., Comment: 10 pages. Code will be released soon

Oriented object detection in aerial images is a challenging task as the objects in aerial images are displayed in arbitrary directions and are usually densely packed. Current oriented object detection methods mainly rely on two-stage anchor-based detectors. However, the anchor-based detectors typically suffer from a severe imbalance issue between the positive and negative anchor boxes. To address this issue, in this work we extend the horizontal keypoint-based object detector to the oriented object detection task. In particular, we first detect the center keypoints of the objects, based on which we then regress the box boundary-aware vectors (BBAVectors) to capture the oriented bounding boxes. The box boundary-aware vectors are distributed in the four quadrants of a Cartesian coordinate system for all arbitrarily oriented objects. To relieve the difficulty of learning the vectors in the corner cases, we further classify the oriented bounding boxes into horizontal and rotational bounding boxes. In the experiment, we show that learning the box boundary-aware vectors is superior to directly predicting the width, height, and angle of an oriented bounding box, as adopted in the baseline method. Besides, the proposed method competes favorably with state-of-the-art methods. Code is available at https://github.com/yijingru/BBAVectors-Oriented-Object-Detection., Comment: Accepted to WACV2021

Cosmic metals are widely believed to be produced by supernovae (SNe) and compact object mergers. Here, we discuss the nucleosynthesis of neutrino-dominated accretion flows (NDAFs) with outflows in the core-collapse SNe (CCSNe), and show that the outflows from NDAFs can have a significant contribution to the $^{56}$Ni abundance in the faint explosions if the masses of the progenitor stars are within about $25-50$ $M_\odot$. Less massive progenitor stars can produce more $^{56}$Ni than their more massive counterparts in the NDAF outflow nucleosynthesis channel. Therefore, we find that the total (i.e., CCSNe and NDAF outflows) $^{56}$Ni mass per CCSN depends only weakly upon the mass of progenitor stars. In the metallicity evolution, the ratio of $^{56}$Fe (decayed by $^{56}$Ni) mass to the initial total gas mass can increase by $\sim$ 1.95 times if the upper limits of the nucleosynthesis yields from NDAF outflows and CCSNe are considered. Our results might have significant implication for chemical evolution of the the solar neighborhood, galaxies, and active galactic nuclei., Comment: 10 pages, 5 figures, accepted for publication in ApJ