The combination of Unmanned Aerial Vehicle (UAV) technology and computer vision has become popular in a wide range of applications, such as surveillance and reconnaissance, while popular evaluation measures are sometimes not applicable for specific tasks. In order to evaluate visual object detection and tracking algorithms of low-altitude aerial video properly, we first summarize the evaluation basis of computer vision tasks, including ground truth, prediction-to-ground truth assignment strategy and distance measures between prediction and ground truth. Then, we analyze the advantages and disadvantages of visual object detection and tracking performance measures, including average precision (AP), F-measure, and accuracy. Finally, for the low-altitude (nearly 100 m) surveillance mission of small unmanned aerial vehicles, we discuss the threshold optimization method of popular measures and the design strategy of application measures. Our work provides a reference in the aspect of performance measures design for researchers of UAV vision.