Automatic tracking is an important basis for abnormal behavior management and disease prediction of livestock. In commercial farms, the use of surveillance video to track and monitor dairy goats is conducive to improving production efficiency and commercial welfare. In this paper, an algorithm-based on Siamese strategy is presented for the automated tracking of a single dairy goat in the surveillance video. The Dairy Goat Dataset (DG-dataset) containing 200 dairy goat motion videos with a total of 161,000 frames of images randomly collected from the farm was created, and the ImageNet VID, Youtube-BB, and GOT-10 k were used for training. First, the proposed tracker named SiamBNAN employs an effective and highly modular backbone network constructed by the Multi-Convolution Residual Blocks (MCRBs) and Down-sampling Multi-Convolution Residual Blocks (D-MCRBs). The MCRBs and D-MCRBs replace the original square convolution kernel with a kernel augmented by asymmetric convolution kernels and perform a set of group convolutions. Finally, the Regional Proposal Network (RPN) is used for foreground-background classification and proposal regression. The experimental results show that this algorithm outperforms SiamFC, SiamRPN, and SiamRPN + in terms of both Expected Average Overlap (EAO), Robustness (R), Success Rate (Succ), and Precision (Prec) on the DG-dataset. The SiamBNAN runs at 70 fps with low computing space requirements showing that it is effective and could be used for monitoring the behavior of dairy goats in the real farming scene.