As the combination of foundation pile and geothermal heat exchanger, the energy pile has been attracting its industry application and research interests. For the heat transfer model, different analytical models have been established to describe the heat transfer process of the energy pile, e.g. the solid-cylindrical source model, the ring-coil source model, and the spiral heat source model. Most of these analytical models assumed that the pile and the ground soil share the same thermos-physical properties. However, the thermos-physical properties vary with the types of soil, and large property difference may exist between piles and soil. Ignoring this difference, severe calculation error may occur in the heat transfer estimation. Therefore, this paper will establish a numerical model to investigate and examine the impact of thermo-physical properties on the heat transfer performance of energy piles. Different types of soil materials are simulated, and the simulation results show that the assumption of a homogenous domain (in most of the available analytical models) may lead to an incorrect estimation, especially in short-term of operation. Assuming the heat capacity increasing from 1600 to 4800 kJ/K, a temperature difference of 7.16 K with an error of 46.5% can be obvious after 10 days of operation. Also, the inaccuracy would be larger when the pile is larger in radius. So, it can be concluded that differentiating the thermo-physical properties of the pile and soil in calculating the short time temperature response of an energy pile is necessary, especially when the radius ratio of the heat exchanger to the pile is small.