• The subspace iteration method is used to solve nonlinear eigenvalue problem. • The continuation method is used to solve the nonlinear Hermitian eigenproblem. • The method presented allows determination of only a part of eigenvalues. • The method was tested for structures with viscoelastic elements. • The models of VE elements are described with the help of fractional derivatives. The paper presents an extension of the subspace iteration method for application in systems with viscoelastic damping elements, which are described by both classical and fractional models. The presented method enables determination of only a certain number of eigenvalues and associated eigenvectors. This is very useful, especially when the considered problem has many degrees of freedom, in which case it would be very time-consuming or even impossible to determine all the eigenvalues. At the same time, from the engineering point of view, it is not necessary to know all the eigenvalues. In this paper, the subspace iteration method is used for the first time to solve the nonlinear eigenproblems which appear in the dynamic analysis of systems with viscoelastic damping elements. The proposed solution consists of assuming the number of eigenvalues to be determined, taking the initial point of iteration and then solving the nonlinear reduced eigenproblem with Hermitian matrices in each iterative loop. The solution to the reduced eigenproblem is obtained using the continuation method, and it is an additional novelty in the paper. The correctness and effectiveness of the proposed approach are illustrated with numerical examples. [ABSTRACT FROM AUTHOR]