Search

Sparsely-gated mixture-of-experts (MoE) has been widely adopted to scale deep learning models to trillion-plus parameters with fixed computational cost. The algorithmic performance of MoE relies on its token routing mechanism that forwards each input token to the right sub-models or experts. While token routing dynamically determines the amount of expert workload at runtime, existing systems suffer inefficient computation due to their static execution, namely static parallelism and pipelining, which does not adapt to the dynamic workload. We present Flex, a highly scalable stack design and implementation for MoE with dynamically adaptive parallelism and pipelining. Flex designs an identical layout for distributing MoE model parameters and input data, which can be leveraged by all possible parallelism or pipelining methods without any mathematical inequivalence or tensor migration overhead. This enables adaptive parallelism/pipelining optimization at zero cost during runtime. Based on this key design, Flex also implements various MoE acceleration techniques. Aggregating all techniques, Flex finally delivers huge speedup at any scale -- 4.96x and 5.75x speedup of a single MoE layer over 16 and 2,048 A100 GPUs, respectively, over the previous state-of-the-art. Our evaluation shows that Flex efficiently and effectively runs a real-world MoE-based model named SwinV2-MoE, built upon Swin Transformer V2, a state-of-the-art computer vision architecture. On efficiency, Flex accelerates SwinV2-MoE, achieving up to 1.55x and 2.11x speedup in training and inference over Fairseq, respectively. On effectiveness, the SwinV2-MoE model achieves superior accuracy in both pre-training and down-stream computer vision tasks such as COCO object detection than the counterpart dense model, indicating the readiness of Flex for end-to-end real-world model training and inference.

Machine learning models made up of millions or billions of parameters are trained and served on large multi-GPU systems. As models grow in size and execute on more GPUs, the collective communications used in these applications become a bottleneck. Custom collective algorithms optimized for both particular network topologies and application specific communication patterns can alleviate this bottleneck and help these applications scale. However, correctly and efficiently implementing custom algorithms is challenging. This paper introduces GC3, a system for programmable GPU communication. GC3 provides a domain specific language for writing collective communication algorithms and an optimizing compiler for lowering them to an executable form, which can be executed efficiently and flexibly in an interpreter based runtime. We used GC3 to write novel collective algorithms for AllReduce and AllToAll that are up to $1.9\times$ and $1.3\times$ faster than hand-optimized implementations, respectively.

This paper focuses on the construction of urban rail transit projects to prevent the risk of water inrush disasters faced during subway construction in cities with abundant groundwater and...

The karst development of Jiao Dong Peninsula in Shandong Province has formed a unique karst landform, and has become one of the typical karst areas in the north of China. Karst collapse occurred during the construction of high-speed railways and caused many environmental problems. To reveal the development situation of karst area in Shandong, the conventional surface geophysical techniques were used for karst exploration and location of key area. And the results of drilling verification were in good agreement with the results of geophysical exploration. At the same time, the new technology in the field of surveying and mapping 3D laser scanning technology, is cited in the engineering geophysical exploration, and the key issue is the shape and size of the karst. It makes up for the shortcomings of traditional geophysical exploration technology that can only be qualitative and cannot be quantitatively analyzed. It realizes the transformation of karst exploration from regional detection to single cave exploration, and better implements the concept of information construction. It has contributed to the smooth progress of the project and environmental protection.

The angiopoietin-1 (Ang-1)/Tie-2 signaling pathway plays a crucial role in the maintenance of vascular stabilization and permeability. In this study, we evaluated the protective effect of a designed Ang-1 variant (COMP-Ang-1) on peritoneal vascular permeability and peritoneal transport function in a uremic peritoneal dialysis (PD) model. Compared to the sham controls, uremic rats were characterized by decreased pericyte coverage and downregulated endothelial junction expression. The permeability of the peritoneal vasculature to FITC-BSA and FITC-dextran in uremic rats was also higher than that in the sham controls, as well as increased levels of proinflammatory adhesion molecules and cytokines, increased D/P(cr) and decreased ultrafiltration. Such changes were more marked in uremia+PD rats after exposure to glucose-based peritoneal dialysis fluid (PDF) for 4 weeks. Peritoneal Ang-1 protein expression and Tie-2 phosphorylation were significantly lower in uremic rats than in control rats and were further significantly reduced in uremia+PD rats. After COMP-Ang-1 administration, phosphorylation of the Tie-2 receptor was significantly increased. Treatment with COMP-Ang-1 also significantly enhanced pericyte coverage, upregulated endothelial junction protein expression and inhibited leakage of FITC-BSA and FITC-dextran from the peritoneal vasculature induced during PD therapy; these changes were accompanied by reduced peritoneal tissue levels of proinflammatory adhesion molecules and cytokines, decreased D/P(cr) and increased ultrafiltration. These findings suggest that COMP-Ang-1 may exert a protective effect against glucose-based PDF-induced peritoneal vascular permeability and inflammation, at least in part, by enhancing pericyte coverage and endothelial junction protein expression, which subsequently significantly improves peritoneal transport function.

This paper considers that the volumes of loading and unloading at the hub port and feeder port during the downturn shipping period obey the interval distribution. An optimum design model is established by using voyage cost minimum as object, assuming different ship sail speed of upstream and downstream, ship type, including interval variables with hard time windows of route planning and scheduling model of the shipping linear. As the difficulty of solving the uncertain variables in the model, an interval number programming method based on fuzzy constraint satisfaction is used to convert the uncertain part of the original uncertain model into a deterministic mixed integer linear programming model, and a suitable encoding scheme is designed to improve the traditional genetic algorithm to solve the model. A shipping company is taken as an example to verify the effectiveness of the model and optimization algorithm. The results show that this method can solve the problem of how to allocate the route and the port order of scheduling, and reduce the cost for shipping companies.