1. System performance optimization practice for big data scenarios
- Author
-
Jibin WANG, Hailong YANG, Kai FENG, Xin SUN, Minda ZHANG, Kelun LEI, Zhiwen XIAO, Yifei ZHANG, and Jiaxi WU
- Subjects
Hadoop ,big data system ,performance optimization ,tuning tool ,Electronic computers. Computer science ,QA75.5-76.95 - Abstract
In the existing large-scale distributed environments, there is still much room for improvement in the performance and computational efficiency of big data applications.However, performance analysis and optimization in large-scale environments requires a large number of human resources from domain experts.This paper proposes a general lowperformance query statement detection and optimization process for performance optimization in big data applications, summarizes four types of low-performance behaviors that significantly affect the performance of big data applications, and proposes specific optimization strategies for each type of low-performance behavior.Finally, through experimental evaluation, the effectiveness of the optimization scheme in actual large-scale cluster is verified.
- Published
- 2024
- Full Text
- View/download PDF