The background of gravitational waves produced by the ensemble of rotating neutron stars (which includes pulsars, magnetars, and gravitars) is investigated. A formula for Ω(f) (a function that is commonly used to quantify the background, and is directly related to its energy density) is derived, without making the usual assumption that each radiating system evolves on a short time scale compared to the Hubble time; the time evolution of the systems since their formation until the present day is properly taken into account. Moreover, the formula allows one to distinguish the different parts of the background: the unresolvable (which forms a stochastic background or confusion noise, since the waveforms composing it cannot be either individually observed or subtracted out of the data of a detector) and the resolvable. Several estimations of the background are obtained, for different assumptions on the parameters that characterize neutron stars and their population. In particular, different initial spin period distributions lead to very different results. For one of the models, with slow initial spins, the detection of the background by present or planned detectors can be rejected. However, other models do predict the detection of the background, that would be unresolvable, by the future ground-based gravitational wave detector ET. A robust upper limit for the background of rotating neutron stars is obtained; it does not exceed the detection threshold of two cross-correlated Advanced LIGO interferometers. If gravitars exist and constitute more than a few percent of the neutron star population, then they produce an unresolvable background that could be detected by ET. Under the most reasonable assumptions on the parameters characterizing a neutron star, the background is too faint to be detected. Previous papers have suggested neutron star models in which large magnetic fields (like the ones that characterize magnetars) induce big deformations in the star, which produce a stronger emission of gravitational radiation. Considering the most optimistic (in terms of the detection of gravitational waves) of these models, an upper limit for the background produced by magnetars is obtained; it could be detected by ET, but not by BBO or DECIGO. Simple approximate formulas to characterize both the total and the unresolvable backgrounds are given for the ensemble of rotating neutron stars, and, for completion, also for the ensemble of binary star systems. [ABSTRACT FROM AUTHOR]