Maize residue detection is an essential factor for conservation tillage; however, traditional methods are not robust for complex environments. Deep neural networks can learn multi-scale features from large datasets; nevertheless, it is difficult to obtain sufficient labelled maize residue data. Hence, it is important to develop an approach that can realize maize residue segmentation from few-shot samples. In this paper, a Siamese domain transfer network (SDTN) architecture is proposed to transfer convolution features to maize residue segmentation. First, an intermediate domain is used to bridge the distance between the source and target domains, which is a typical limiting factor in domain transfer. Next, the knowledge is transferred via the SDTN, in which hidden convolution layers represent multi-scale feature maps. Specific layers are explicitly matched using different domain distributions with designated confusion distances. Finally, the network is annealed and trained with few-shot samples. The experiments indicate that our transfer strategy and architecture can effectively transfer features and achieve a segmentation error of 33.2 AP at 5 fps. Furthermore, our model can reduce the error obtained by direct fine-tuning without domain transfer by 4.6%.