Search

In this study, we introduce a novel training algorithm specifically designed to overcome the limitations of GPU memory on a single DGX-A100 system. By utilizing the CPU and main memory in the training process and applying a strategy of division and parallelization, our algorithm enhances the size of the trainable language model and the batch size. In addition, we developed a comprehensive management system to effectively manage the execution of the algorithm. This system systematically controls the training process and resource usage, while also enabling the asynchronous deployment of tasks. Finally, we proposed a scheduling technique integrated into the management system, promoting efficient task scheduling in a complex, heterogeneous training environment. These advancements equip researchers with the ability to work with larger models and batch sizes, even when faced with limited GPU memory.

Recently, language models based on the Transformer architecture have been predominantly used in AI natural language processing. These models, which have been proven to perform better with more parameters, have led to a significant increase in model size and computational load. ALBERT solves this problem by significantly reducing the number of parameters it retains by repeatedly reusing parameters. Although ALBERT significantly reduces the parameters it maintains, it requires a computational load similar to the original language model due to the reuse process. In this study, we develop a distillation system that decreases the number of times the ALBERT model reuses parameters and progressively reduces the parameters being reused. We propose a representation in this distillation system that can effectively distill the knowledge of the original model and develop a new architecture with reduced computation. Through this system, F-ALBERT, which had about half the computational load compared to the ALBERT model, restored about 98% of the performance of the original model on the GLUE benchmark.

This paper presents an experiment-based synthetic structural analysis method that combines digital image processing (DIP) and the particle difference method (PDM), which is a strong form-based meshfree method. The proposed method uses images to determine the displacement of deformed specimens, interpolates the displacement onto nodes of the PDM model without meshes or grids, and calculates the kinematic variables. Furthermore, the pixel extraction method for the target area and the method of setting the region of interest for expediting DIP were used during the synthetic structural analysis. A method for effectively expanding the number of tracking points and an improved method for labeling tracking points are also presented. To verify the performance of the analysis, the experimental and numerical analysis results of a three-point bending test on a rubber beam were compared in terms of various mechanical variables as well as with the PDM results of a simulated bending test. It was found that tracking point expansion and adjusting the radius of the domain of influence are advantageous for performing an accurate calculation without losing computational efficiency. It was demonstrated that the synthetic structural analysis effectively overcomes the shortcomings of the conventional experiments and the limitations of pure simulations.