Particle filter algorithms are widely used for object tracking in video sequences, but the standard particle filter algorithm cannot solve the validity of particles ideally. To solve the problems of particle degeneration and sample impoverishment in a particle filter tracking algorithm, an improved object tracking algorithm is proposed, which combines a multi-feature fusion method and a genetic evolution mechanism. The algorithm dynamically computes the feature's fusion weight by the discriminability of each vision feature and then constructs the important density function based on selecting a feature's fusion method adaptively. Moreover, a self-adaptive genetic evolutionary mechanism is introduced into the particle resampling process and makes the particle become an agent with the ability of dynamic self-adaption. With self-adaptive crossover and mutation operators, the evolution system produces a large number of new particles, which can better approximate the true state of the tracking object. The experimental results show that the proposed object tracking algorithm surpasses the conventional particle filter on both robustness and accuracy, even though the tracking object is very challenging regarding illumination variation, structural deformation, the interference of similar targets and occlusion.