Mutation takes a vital part in assisting differential evolution (DE) to achieve satisfactory performance. The most crucial factor for a good mutation scheme is to mutate individuals dispersedly but with fast convergence to optimal regions. With this purpose, this paper designs a novel mutation approach, termed as “DE/current-to-gselite/1”, by utilizing the Gaussian distribution to sample guiding exemplars around elites in the population to evolve individuals. Accordingly, a Gaussian sampling guided differential evolution (GSGDE) is devised to hopefully tackle optimization problems effectively. With the assistance of the Gaussian distribution, GSGDE mutates distinct individuals with very different guiding exemplars. Hence, high mutation diversity is expectedly maintained, which leads to that individuals could traverse the problem space in diverse directions. Thanks to the narrow sampling range of the Gaussian distribution, the generated guiding exemplars are likely better and thus individuals in the population are anticipated to move towards optimal regions fast. This is of great profit for fast convergence to high-quality solutions. Further, a dynamic parameter adjustment strategy is proposed to dynamically regulate the number of elites. Hereafter, GSGDE gradually shifts from concentrating on exploring problem space to focusing on exploiting found optimal areas. Cooperated with an existing adaptive parameter strategy, GSGDE is anticipated to strike a good balance between exploitation and exploration to traverse the problem space and hence likely obtain satisfactory performance. Experiments have been extensively carried out on the latest CEC2014 and CEC2017 problem suites with three settings of the dimensionality. Experimental results substantiate that GSGDE has a good scalability and attains highly competitive performance with or even significantly superior performance to 11 latest and representative DE methods. Particularly, its superiority becomes more and more significant as the dimensionality increases.