1. An Energy Efficiency Optimization and Control Model for Hadoop Clusters
- Author
-
Yunpeng Cao and Haifeng Wang
- Subjects
wavelet neural networks ,General Computer Science ,Computer science ,Control (management) ,Hadoop cluster ,General Engineering ,Computational science ,minimizing energy consumption ,model prediction control ,energy efficiency control ,General Materials Science ,DVFS ,lcsh:Electrical engineering. Electronics. Nuclear engineering ,lcsh:TK1-9971 ,Computer Science::Distributed, Parallel, and Cluster Computing ,Computer Science::Databases ,Efficient energy use - Abstract
The majority of large-scale data intensive applications designed by MapReduce model are deployed and executed on a large-scale distributed Hadoop system. Running such application on a large cluster requires a large amount of energy. Therefore improving energy efficiency and minimizing energy consumption when executing each MapReduce job is a critical concern for data centers. We propose a control model based on model prediction control (MPC) for improving energy efficiency of Hadoop cluster while satisfying performance goal. The controller can dynamically scale CPU frequency in facing the workloads changes. The wavelet neural network is utilized to build the prediction model, which can accurately estimate energy efficiency ratio based on the features extracting from the variation of Hadoop workloads. With the help of a wavelet neural network, the controller can guarantee the energy efficiency ratio converges to an ideal set point within a finite settling time. We perform the extensive experiments on a Hadoop cluster to compare our controller with the previous MPC controller. The results demonstrate that our controller outperforms the baseline on energy efficiency control and adapts to workloads of Hadoop cluster.
- Published
- 2019
- Full Text
- View/download PDF