Background and Aims Evolutionary transitions between separate and combined sexes have frequently occurred across various plant lineages. In mosses, which are haploid-dominant, evolutionary transitions from separate to combined sexes are often associated with genome doubling. Polyploidy and hermaphroditism have strong effects on the inbreeding depression of a population, and are subsequently predicted to affect the mating system. Methods We tested the association between ploidy (haploid, diploid or triploid gametophytes) and mating system in 21 populations of Atrichum undulatum sensu lato, where sex ratios vary widely. For each population, we measured the sex ratio, estimated selfing rates using allozyme markers and determined the level of ploidy through flow cytometry. Key Results Hermaphrodites in A. undulatum were either diploid or triploid. However, many diploid populations were strictly separate-sexed, suggesting that hermaphroditism is not a necessary result of genome doubling. Levels of selfing were strongly supported as being greater than zero in one population with strictly separate-sexed individuals, and one-third of populations with hermaphrodites. Conclusions Although hermaphrodites are associated with triploidy, hermaphroditism is not a necessary outcome of genome duplication. Hermaphroditism, but not genome duplication alone, increased estimated selfing rates, probably due to the occurrence of selfing within a gametophyte. Thus, genome duplication can influence the mating system and the associated evolution and maintenance of reproductive traits. [ABSTRACT FROM AUTHOR]