Wolframite image recognition is an efficient way to replace the concentrator for manual sorting, but the problem is that wolframite and the surrounding rock cannot be recognised. In this paper, convolutional neural network in deep learning was used to solve the problem. This method is fast convergence, small data set and accurate classification. Firstly, the RGB image of wolframite was augmented by rotation and translation to reduce sample imbalance. Secondly, the neural network optimised in this paper is used for new training based on Keras framework. Finally, the results show that Wu-VGG19 has the highest recognition rate of 97.51% in wolframite and surrounding rock recognition. In addition, a quartz gangue category is added to continue the experiment, and the final result shows that the improved inception Wu-v3 has the highest ore recognition rate, 99.6%., Wolframite image recognition is an efficient way to replace the concentrator for manual sorting, but the problem is that wolframite and the surrounding rock cannot be recognised. In this paper, convolutional neural network in deep learning was used to solve the problem. This method is fast convergence, small data set and accurate classification. Firstly, the RGB image of wolframite was augmented by rotation and translation to reduce sample imbalance. Secondly, the neural network optimised in this paper is used for new training based on Keras framework. Finally, the results show that Wu-VGG19 has the highest recognition rate of 97.51% in wolframite and surrounding rock recognition. In addition, a quartz gangue category is added to continue the experiment, and the final result shows that the improved inception Wu-v3 has the highest ore recognition rate, 99.6%.